Learning Outcomes of an "Astronomy/STEAM" Blended Learning Program with VR/AR experience
Creators: name: Dr. Vrioni, Andri
name: Dr. Neocleous, Kleanthis
Publisher: name: Zenodo
Year: 2024

description: Learning Outcomes of an "Astronomy/STEAM" Blended Learning Program with VR/AR experience. 

 

University of Nicosia

Vrioni.a@unic.ac.cy



Alexandros Polidorou, Fotos Frangoudes and Dr Kleanthis Neokleous Vasiliki Panagi, 

Kleanthis Neokleous

 

Abstract 

As part of participants’ experience astro-parks, astro-camps and astro- village, the astronomy/ STEAM educational blended learning program also including VR/AR experience integrating astronomy into STEAM., developed and validated. The purpose of this study was to investigate the learning outcomes of the program. Two sources of information for results triangulation, give input to the program’s effectiveness. The first source of information is teachers’ assessments with the use of Rubric based on students’ observed behavior and products during the implementation of the program. The second source of information is from students’ perspective as measured with questionnaire including four dimensions: self-efficacy, learning satisfaction, learning motivation, and learning effectiveness. Thirty-two (32) students 12y-18y participated in the program. Both the Repeated Measures ANOVA analysis and the students’ self-assessment analysis results confirms significantly improved results comparing pre- and post- learning measurements and indicate that the program offered was effective in producing learning to students with regards the four dimensions examined.  

 

Keywords: Rubric, 21st Century Skills, STEAM Knowledge and Interest, STEAM learning outcomes

 

Introduction

The nature of Learning in blended and in-formal learning settings

 Everyday learning includes a range of experiences that may extend over a lifetime, such as personal hobbies, watching a movie, reading books or magazines, surfing the Web, or participating in camps. These experiences are very much selected and shaped by the learners themselves and may vary greatly across families, communities, and cultures. People engaging in everyday learning may not be aware that they are learning. Instead, they simply see the activity as part of their daily lives. Consequently, learners may not be explicitly asked to demonstrate competence in the same way they are when tested in school (Stroud, Groome, Connolly, & Sheppard, 2007). Despite the diversity of thought in the informal STEAM learning community about what outcomes are most important and what approaches to measuring them are most appropriate, there is an emerging consensus on core assumptions regarding the nature of outcomes in informal learning. Outcomes can be based on the goals and objectives of the nature of each program, or they can be unplanned and unanticipated, based on what individual learners find to be most valuable. 

Learning outcomes in informal settings as is the participation in an astrocamp, are different from traditional formal learning. Even though traditional learning is structured and targeted on knowledge cultivation, the learning outcomes are not always as expected mostly because traditional learning settings neglect emotional aspects of learning, authenticity, multimodality, interactivity (Vrioni, 2016). Students a little after a traditional learning experience tend to forget what they learn (Vrioni, 2016). It is becoming increasingly evident that the impact of informal learning experiences, can last over time (National Research Council. (2009). Participants in an informal STEAM learning program, for example may remember for years after a visit or experience they received during their interaction with a STEAM learning experience in in-formal setting (National Research Council, 2009). 

Today, rapid improvements information and communication technologies influence and change the educational field.  New approaches towards learning and teaching in both formal and informal learning settings. Blended learning is a rotation approach, where participants in groups move through a set of online STEAM learning activities for example, in the physical learning setting with the guidance of a teacher. In other words, in informal settings science centers for example, teacher-led instruction is minimized, and the participants may engage in common tasks online. Blended learning creates a truly integrated learning setting where the needs of all types of learners can be met. Keeping participants engaged, stimulated, and motivated also results effective learning and greater learning gains.  Blended learning environments also secure increased access and convenience.  The nature of learning outcomes in informal settings enhanced with technology (blended) may be: 





Sparking Interest and Excitement




Understanding STEAM Content and Knowledge. Generating, understanding, remembering, and using concepts, explanations, arguments, models, and facts related to science.




Engaging in Processes 




Reflecting on STEAM as a way of knowing, including the processes, concepts, and institutions of Science, Technology, Engineering, Arts and Mathematics. It also involves reflection on the learner’s own process of understanding natural phenomena and the scientific explanations for them.




Participation in STEAM activities and learning practices with others, using relevant language and tools.




Identifying with the STEAM Enterprise. Coming to think of oneself as a STEAM learner and developing an identity as someone who knows about, uses, and sometimes contributes to STEAM. Knowledge and willingness to follow a STEAM career (National Research Council, 2009).



The nature, the goals, and practices of such learning demand for relevant assessment tools.  (National Research Council, 2009). 

 

Insights from research on STEAM learning

 

The birth of STEAM education was motivated by the highly competitive global challenges of the world (Kim, Chu, & Lim, 2015). Traditional teaching and learning practices are ineffective to cultivate scientific literacy and high order thinking skills as indicated by several relevant research (Organization for Economic Co-operation and Development-OECD, 2018). In recent years, there is an emphasis in the wider educational research community as well as across many nations on an inclusive for all STEAM (Science, Technology, Engineering, Arts and Mathematics) education. This will support the growth of a skilled STEAM workforce of tomorrow. Setting up STEAM in the classroom is about preparing students to express innovative and creative ideas and taking ownership of their learning. It’s about becoming curious about the world around them and be empowered to change it for the better. 

 

Astronomy is the science that studies the origin and evolution of the Universe and everything in it. This definition seems simple, but the Universe is a vast place, filled with fascinating celestial objects of all sizes, shapes and ages, and with amazing phenomena. Astronomy is not just about scientific advances or technological applications: it gives us the opportunity to broaden our limited horizons, discover the beauty and grandeur of the Universe and our place in it. This view, commonly referred to as “the cosmic perspective”, is one of the most important contributions of Astronomy to Humanity.  Several texts (e.g., Adams & Slater, 2000; Stahly, Krockover, & Shepardson, 1999) mention the scarcity of research in astronomy education. Limited aspects of astronomy have been popular topics in school curricula for decades. In the lower primary school, themes such as “space” or “the planets” have been used by teachers to inspire and fascinate students in much the same way as dinosaurs have captivated small children. Even more limited is the astronomy activities integrated in a STEAM context and following a curriculum to support astronomical literacy (Vrioni, 2022). 

 

STEAM learning can be very effective if builds upon the essential elements of interactivity, multimodality and authenticity are assured when a learning activity a) follows the Problem/ project -Based Learning (PBL) methodology b) authentic assessment and c) adopts the Universal Design of Learning (UDL) offered as a blended learning experience (Vrioni, 2019 a, b, c, d, e, 2022). 

 

PBL provides methodological support for the cultivation of 21st century skills (Barrows, 1985; Kolodner et al., 2003; Rotherham & Willingham, 2009; Sawyer, 2005; Snape & Fox-Turnbull, 2011; Thomas, 2000; Noweski et al., 2012;). Students are given real-world questions/ scenarios related to their area of study (DiMartino & Castaneda, 2007). PBL learning focuses on solving authentic, ill-structured problems in a case study context (Barrows, 1985; Barrows, 2000; Ke, 2014), conduct research, integrate theory, and practice, and apply knowledge and skills to develop a viable solution to a problem, develop a project, design artifacts (Ke, 2014), create an authentic product (Bell, 2010) and reflect on the learning experiences (Hmelo-Silver & Barrows, 2008). 

 

Authentic Assessment is a new notion supporting STEAM educational context. Authentic assessment holistically and continuously records students’ achievement, including both product and learning process and is inseparable from teaching and learning process. Authentic assessment is something more than formative assessment (Boud, 2000). A tool to support authentic assessment is “Rubric”. A “Rubric” is an established set of criteria used for evaluating or rating student products and performances (Ronis, 2008). “Rubrics” contain three essential features: criteria students are to attend to in completing the assignment, markers of quality (typically rating scales), and scoring. Criteria are used in determining the level at which student work meets expectations. Markers of quality give students a clear idea about what must be done to demonstrate a certain level of mastery.  Reviewing the work of Reddy & Andrade, (2010); Audrey, (2006); Vrioni, Mavroudi, Ioannou, (2019); Vrioni & Ioannou, (2021) it is highlighted that authentic assessment with the use of rubrics helps students to focus their efforts on completing assignments in line with clearly set expectations, connect the lessons with their experience, real world, become motivated and monitor their progress. 

 

UDL is an educational framework that encompasses the requirements of inclusion aiming at equal access to Education, without exclusions. It seeks to create flexible learning environments that bridge individual differences and nurture physical, cognitive, mental, and organizational barriers. It focuses on the context of STEAM education on the redesign of curricula (objectives, assessment, methods, and materials) for the inclusion of all students. The UDL framework is defined by a set of principles and guidelines that ensure accessibility and the reduction of learning barriers. Their existence is a key element of flexibility and serves to be flexible and applicable in any learning environment. These principles relate to interventions in space and the environment, the classroom environment, materials, interaction and action, evaluation, and feedback. Thus, the UDL is organized around three basic principles: a) The provision of multiple ways of presenting information. b) The provision of multiple ways of action for the expression of acquired knowledge. c) The provision of multiple ways of involvement and motivation for learning. 

 

Insights from research on VR/AR learning

Studies have shown that using VR/AR technology in STEAM/Astronomy education increases student engagement and motivation to learn (Wu et al., 2021; Li et al., 2020). This is because VR/AR technology provides an immersive and interactive learning experience, which is more engaging than traditional classroom teaching.

 

VR/AR technology has also been shown to improve retention and learning outcomes in STEAM/Astronomy education (Makransky et al., 2019; Wu et al., 2021). This is because VR/AR technology allows students to visualize and interact with complex concepts and phenomena, which enhances their understanding and retention of the material.

 

Studies have shown that VR/AR technology can increase gender and ethnic diversity in STEAM/Astronomy education by providing an inclusive and equitable learning environment (Liu et al., 2020; Zhou et al., 2019). This is because VR/AR technology allows students to learn in a self-paced and personalized manner, which accommodates different learning styles and abilities. 

 

VR/AR technology can also be cost-effective and scalable in STEAM/Astronomy education, as it can reduce the need for expensive and complex equipment and materials (Makransky et al., 2019; Wu et al., 2021). This makes STEAM/Astronomy education more accessible to a wider range of students and educational institutions.

 

Purpose of the Study 

The purpose of this study was to investigate the learning outcomes of an astronomy/ STEAM blended learning program also including VR/AR experience. The program implemented during its pre-pilot testing phase in a makerspace, before its pilot testing at astro-parks/ astro-camps/ astro-villages, working in the same conditions with VR/AR blended learning delivery, accessed online material via smartphones and use of recycled materials for students’ “engineering” products. Two sources of information for results triangulation, give input to the program’s effectiveness. The first source of information is teachers’ assessments with the use of STEAM Learning Rubric as developed by Vrioni and Ioannou, 2019) based on students’ observed behavior and products during the implementation of the program. The second source of information is from participants’ perspective on their self-efficacy, motivation, satisfaction, and effectiveness as measured with questionnaire including four dimensions: self-efficacy, learning satisfaction, learning motivation, and learning effectiveness.

 

The astronomy/ STEAM program

Within the framework of the implementation of the "Astrotourism in Cyprus" Program, INTEGRATED / 0918/0038, co-financed by the European Regional Development Fund and the Republic of Cyprus through the Foundation for Research and Innovation, the "Astro- camps" will operate, among other actions promoting astrotourism in Cyprus. Astro-tourism is a new travel trend that Cyprus is beginning to embrace. The idea is that tourists can add to their island experience by stargazing at night. The experience is at its best in the great outdoors where the light pollution of towns and cities doesn’t obscure the stars. Project idea is to create astro-parks, organize astro-camps, set up astro-villages, develop a relevant program and fill it with events. The overall goal is to conduct research that will focus on “beyond the state-of-the-art” innovation in the field of astrotourism leading Cyprus to be added on the world map of the astrotourism “hot” destinations and creating the proper environment to host and offer astronomy experiences for both experts and enthusiasts at an entry level.

 

As part of astro-parks, astro-camps and Astro village experience participants will have the experience of a. VR/AR astronomy experience b. the astronomy/ STEAM program, an educational program integrating astronomy into STEAM. The program is addressing two challenges: give dedicated care for the environment and offer a high-quality learning experience to the participants. Participants can expand his/her astronomical and overall STEAM knowledge, their interest and motivation and even develop some 21st century skills. The mobile nature of astro-parks/ astro-camps/ astro-villages and special characteristics of the project demanded for flexibility, paperless work, avoidance of heavy equipment. For assuring these astronomy/ STEAM program, developed on Moodle supporting a flexible and paperless enhanced learning experience available to participants in a blended form. Participants will be physically placed at the open space area of mobile astro-parks/ astro-camps/ astro-villages having a face-to- face learning experience supported with smart phones. Program uses a Learning Management System (LMS) for its design and delivery phase. LMS are web-based systems that allow instructors and/or students to share materials, submit and return assignments, and communicate online. Moodle is a popular LMS. It is a free web application that educators can use to create effective online learning sites. Moodle is also available via smart phones.

 

Program’ Moodle page include introductory information: General information (about STEAM, the theoretical background of the program, adding astronomy to STEAM, innovation in the design of the program, process for program development, guidelines for using the learning platform), glossary and digital Library. Program’ Moodle page also include the main program body, organized in fifteen “challenges” in the form of a learning scenario with 3 hours duration each. Each Challenge represents and includes expected learning outcomes, learning materials, supported materials, and learning activities. Participants can access the course material via their smart phones.

 

Program created to engage participants in multiple ways, including physically, emotionally, and cognitively. Program design phase started with the theoretical framework supporting the program the decision of the curriculum (what of learning) and the decision of the Design and validation process of the program. The program design follows the effective STEAM learning practices incorporating the PBL methodology, uses authentic assessment and adopts the UDL model (Vrioni, 2022).  

 

Each Challenge starts always with “Exploring alternative ideas phase”. To address the misconceptions, we first need to identify what misconceptions the students hold. This is achieved with the development of concept cartoons. After showing cartoon to the students, allow students to talk and discuss which student they think gave the correct explanation and give their own reasoning as to why they think that. Then the “Defining the problem” phase, which include the presentation of the challenge to students. A problem that is realistic, complex, and related to everyday life is presented for example: Light pollution. Teacher presents the challenge through multimedia, posters, and articles. Participants login to program material via smartphones and choose for themselves the source they prefer to interact with. The “Define of the process, the products and the quality indicators” step then follow. Teacher at this phase coordinate a discussion to decide the process to follow, the deliverables, the timeframes, the limitations, and specific information with regards the requested task. The assessed criteria of quality are set at this stage. This has to do with the product characteristics but also for procedure of work and specific behavior characteristics especially with regards group work behavior, presentation of work, conflict resolution etc. This is the stage where students define the indicators to be included in the assessment rubric to be used. “Group formation and assignment of the students' roles”, then starts. During this phase groups are formed and specific students’ roles in the project. Most common roles are researchers, presenters, designers, engineers, team coordinator etc. The “Research/Investigation” phase follows. Students, in this phase, further search for resources and interact with relevant mobile apps. At this stage, as well there is a great deal of flexibility in what sources students will use for their research with the use of their smartphones. Next, the “Create” phase begins. Participants decide the several options they have for their team’ products (e.g., with inspiration from the universe they design and develop a product). Students also decide the materials (recycled materials etc.) and available in smartphones software to work with (e.g., moviemaker, tinkercad, etc.), depending on the challenge, the group member special knowledge and skills. There is flexibility on students’ products. The phase “Present your work” then follows. In this phase, the students’ groups will explain how they worked along the process and demonstrate their final products. They are reminded with the “secrets of effective presentation/ communication.” based on which they will self, and peer evaluate their work. This is the time for the evaluation phase. During the “Evaluation” phase, students self-reflect on their work and the work of others and based on given indicators, they evaluate their works; they decide future revision of their work, their practices, or behaviors. Embedded in this phase it is the “Metacognitive activities” phase, where students reflect on learning procedure, identify what they learn, set goals, and monitor their progress.

 

Program Curriculum Design and validation process

 

The design and validation procedure started with receiving feedback from learning stakeholders with regards their expectations from an effective Astronomy STEAM program. To this end, two (2) scholars, two (2) teachers, two (2) students and two (2) parents participated in a focus group. Based on learning stakeholders’ feedback and the review of literature findings development team decided a) the learning model b) the learning methodology/delivery mode and c) developed on Moodle the first version of the astronomy/team program. Implementation of a Delphi methodology was necessary to facilitate the astronomy/STEAM learning activities validation process. The Delphi method is based on the basic assumption that the views of a group are generally more accurate than those held by individuals; it is based on what we would call "collective wisdom". The goal of the Delphi method is to gain the consent of a group of "experts" or “stakeholders”. Involving learning stakeholders in the development of learning environments it is a good practice as mentioned by many researchers (Roth, Lawless, Tobin & Zimmerman, 2002). The same group of "learning stakeholders" used in focus groups was set up to provide their critical comments and suggestions for improving the program. After three rounds of incorporating their suggestions in improved versions of the program, the procedure ended. A second version of the program formed as a result of this process. The pre- pilot testing phase at the makerspace then followed. Addressing the issue of the need of a concrete educational program for supporting astronomical literacy the program developed. A range of different aspects of astronomy, including the historical, philosophical, sociological, theoretical, and observational is covered. astronomy/ STEAM program includes a variety of topics from the Earth to the edge of the cosmos and back again. The program covers the essential things that an astronomically literate person should know. 

 

Methodology

 

Data Collection Procedure and Analysis 

The program was implemented from the period of December-February 2022 in fifteen (15) 3-hour sessions. Participants were divided in two age groups. At the first session of the program, teachers of both groups, assessed participants’ initial knowledge and skills (pre-measure). Teachers repeated the same job, at the last (fifteenth) session of the program as (post-measure). Each average performance score considered the student performance scores in all the sub-indicators that were included in the rubric. At the last (fifteenth) session of the program, students also, answered a questionnaire, investigating the STEAM learning based on four dimensions (self-efficacy, learning satisfaction, learning motivation, and learning effectiveness).

 

Rubric for teachers to measure STEAM Learning based on students’ behavior

This research used STEAM Learning Rubric developed by (Vrioni & Ioannou, 2019). STEAM Learning Rubric presented in (Table 1) uses 3 dimensions (A, B, C) to track learning based on students’ behaviour and products. Dimension A includes 6 indicators, Dimension B includes 4 indicators, and dimension C included 3 indicators Each dimension is evaluated based on 4 sub-indicators. Each sub-indicator is described as a student behaviour and is marked with 1 point. In that way, the different scores (ranging from 0 to 4) across the several sub-indicators. The calculation of this grade sySTEAM can give a student grade for each dimension, indicator, and sub-indicator. With respective calculations, we also have final performance score for each student.  

 

Table 1: Rubric Dimensions, Indicators and Sub-indicators








A.21ST CENTURY SKILLS



B. STEAM CAPACITY



C. ACTIVE PARTICIPATION AND INTEREST





A.1 Problem Solving

1. Define the problem 

2. Generate alternative solutions 

3. Evaluation and Select

4. Implement and follow up on the solutions



B.1 Demonstration of STEAM knowledge

1. Demonstrating increase knowledge in specific content areas. 

2. Making connections with everyday world. 

3. Using scientific terminology, 

4. Refer to cognitive conflict experience, how alternative ideas changed as a result of a learning experience



C.1 Active engagement and focus during learning activities.

1. Concentration, persisting in a task or program. 

2. Active engagement, sharing knowledge and ideas. 

3. Expressing enthusiasm, joy, etc.

4. Well-developed interest connection with previous knowledge, experience, learning experience.





A.2 Metacognitive Practices

1. Planning for the task, gathering and organize material

2. Monitor mistakes

3. Evaluate based on criteria

4. Self-reflect



B.2 Development of an understanding of the variety of STEAM careers related to different fields of study.

1. Gaining knowledge about relevant with STEAM professions. 

gaining knowledge of where such jobs and careers exist 

Knowing the Universities that offer these studies.

Connecting with an inspired STEAM related famous person



C.2 Indications for out-of-school engagement

1. Performing STEAM-related activities at home, 

2. Conducting Internet searches for more information; getting books or journals about STEAM; watching TV programs on science

3. Participation in competitions, events etc. 

4. Offer a type of real-life actions related with STEAM





A.3 Creativity

1. Fluency, total number of interpretable meaningful ideas generated in response to stimulus

2. Flexibility as the number of different categories of relevant response

3. Originality

4. Elaboration as the amount of detail in the response



B.3 Application of science process/engineering process/product development process

formulating questions; testing, exploring, predicting, observing, collecting, and analysing data 

demonstrating understanding of the nature of sciences. 

using evidence-based reasoning and argumentation. 

demonstrating engineering design practices/product development procedure



C.3 Active inquiries into STEAM topics, concepts, or practices

1. Exploring ideas verbally or physically. 

2. Questioning, hypothesizing, testing, 

3. Conducting Internet searches for more information; getting books or journals about STEAM; watching TV programs on science

4. Critically analyze the procedure, the results





A.4 Collaboration

1. Define roles

2. Smooth Collaboration

3. Beneficial for the project collaboration

4. Leadership skills



B4. Digital Literacy and other STEAM tools

Demonstrating in class and afterschool records for student assessment:

1. Developing capacity to use scientific instruments/ software/ simulations etc.

2. Developing capacity to use software for data analysis, presentation, communication, research design software, 3d printing,

3. Developing capacity to construct the engineering part of an electronic product

4. Develop capacity for coding





A.5 Communication

1. Listening to the guidelines,

2. Fluency/ body language

3. Amount of information and quality

4. Effective use of software for communication





A.6 Critical Thinking

1. Identify a problem or issue; create inferences on why the problem exists and how it can be solved and collect information or data on the issue through research.

2. Develop and execute solutions.

3. Analyze what solutions worked or did not work.

4. Identify ways to improve the solution.






 

STEAM Learning based on students’ self-assessment.

The questionnaire was developed and improved based on feedback received by two scholars in the education field. Questionnaire included ten multiple-choice questions and one open-ended question. Questions 1-10 evaluate participant’s agreement with a statement. To this end a Likert’s five-point scale, with a range from 1 to 5 and with “strongly disagree” to “strongly agree”, as the evaluation standards. The attitude towards learning setbacks, the achievement of learning goals, the current situation of learning, and the understanding of self-learning conditions. These sub-indicators are considered as self-efficacy and was assessed by Q3, Q4, Q5, Q7, Q9 of the questionnaire. The learning motivation includes three major aspects, namely, value, expectation, and emotion. Learning motivation is one of the important factors affecting learning satisfaction. If the learning content meets the students’ interests and needs, it will improve their learning satisfaction. Learning motivation is assessed by Q2, Q6 of the questionnaire. Learning satisfaction is a feeling or attitude towards learning activities, and if students gain knowledge and expectations in activities that meet their needs, they will get a sense of satisfaction. Learning satisfaction is assessed by the questions Q1, Q8, Q10 of the questionnaire. The learning effectiveness dimension explores how students experience the course on system operation and how they learn to understand personal changes, obtain information, and perform tasks. The learning effectiveness calculated based on the (Q11) of the questionnaire: “What could you say are the benefits from your learning experience?” Students should choose what was true for their case from a list of seven choices: Sparked their interest, understand STEAM content and knowledge, engage in scientific, engineering and design thinking processes, develop their 21st century skills, learn about STEAM careers they can follow.

 

Participants

The participants of this study comprised of: Students: Two (2) students participated as group of stakeholder’s members at focus group and Delphi phase. Thirty-two (32) students divided in two different groups: fifteen (15) students aged 6-10 years and seventeen (17) students aged 13-17 years, participated in the first pilot testing of the program. Teachers: Two STEAM teachers used for a) the implementation of the STEAM learning program and b) kept notes for each student, based on student’ observed behaviour C) conducted the evaluation of learning outcomes in each of the two points in time (i.e., pre- and post- measurement) for each student. STEAM teachers participated in the program are active experienced teachers, holding a bachelor’s degree in some STEAM area, trained in a workshop of 36 hours focusing on Astronomy and STEAM learning and assessment. Two (2) teachers (different from the above) participated as group of stakeholder’s members at focus group and Delphi phase.  Scholars: Two (2) two scholars in the education field participated as group of stakeholder’s members at focus group, questionnaire development and Delphi phase. Parents: Two (2) two parents participated as group of stakeholder’s members at focus group and Delphi phase  

 

Results

STEAM learning outcomes based on students’ behavior 

Table 2 and 3 shows descriptive statistics for pre- and the post-test scores in the three rubric dimensions 21st Century Skills, STEAM Capacity, Active Participation, and Interest. The average and the median scores of students’ performance at the end of the program is higher than those at the end of the first day of the program. A Repeated measures ANOVA analysis 

was conducted with dependent variable students’ observed behavior (learning outcomes) as graded by teachers in each of the two points in time (i.e., pre- and post- measurement) for each student. Time was therefore the independent variable. Repeated measures ANOVA analysis was conducted to identify possible significant differences on students’ pre- post-test performance for the three dimensions, 21stCenturySkills, STEAM Capacity and Active Participation and Interest. For the dimension of 21st Century Skills the analysis showed statistically important differences between the pre- and the post-test total scores F (1,31)=321,82, p≤0.001. For the dimension of STEAM Capacity, the analysis showed statistically important differences between the pre- and the post-test total scores F (1,31)=365,95, p≤0.001. For the dimension of Active Participation and Interest the analysis showed statistically important differences between the pre- and the post-test total scores F (1,31)=200,36, p≤0.001. The analysis for pre- and the post-test total scores showed also statistically important differences F (1,31)=480, 68, p≤0.001.

 








Table 2: Descriptive Statistics for Pre-Test Assessment



 


21stCenturySkills



STEAM Capacity



Active Participation and Interest



 


A1



A2



A3



A4



A5



A6



TOTAL 



B1



B2



B3



B4



TOTAL



C1



C2



C3



TOTAL



 


N= 32





M



1.60



1.38



1.75



1.50



2.12



1.41



9.75



1.63



0.94



1.13



1.31



4.94



1.84



1.13



1.56



4.53





SD



1.34



1.16



0.76



0.76



0.98



0.87



5.54



0.75



0.84



0.98



0.97



3.14



0.85



0.71



0.80



2.12





Table 3: Descriptive Statistics for Post-Test Assessment



 


21st Century Skills



STEAM Capacity



Active Participation and Interest



 


A1



A2



A3



A4



A5



A6



TOTAL 



B1



B2



B3



B4



TOTAL



C1



C2



C3



TOTAL



 


N= 32





M



3.21



2.69



2.88



3.53



3.53



2.66



18.50



2.91



2.56



1.13



2.59



10.50



3.25



2.31



2.69



8.25





SD



0.78



0.64



0.71



0.67



0.85



1.43



5.54



0.93



1.08



1.19



0.91



3.76



0.84



0.96



0.93



2.42






Note: M= Average, SD=Standard Deviation, N=Number of Participants

 

STEAM learning outcomes based on students’ self-assessment.

Students’ self-assessment analysis results indicate that program offered an overall high-quality learning. The self-efficacy questionnaire dimension analysis indicates that all the participants gained an understanding of the learning activities expectations how their performance was evaluated, actively participated in the learning procedure, effectively learn, and had fruitful interactions with the other participants. The learning motivation analysis indicates that all the participants believe that the program learning activities effectively promote their interest on the field and that they feel challenged to learn more. The learning satisfaction analysis indicates that all the participants felt welcome, respected, and pleased with the support received. They also found self-assessment and peer assessment and feedback procedure, during learning, useful for making changes and improvements. The learning effectiveness analysis indicates that all the participants felt that learning experience sparked Interest, increased their understanding in STEAM Content and Knowledge, engaged them in scientific, engineering and design thinking processes, develop several skills and gave inspiration about STEAM careers they can follow. Table 4, present the descriptive statistics and frequencies from students’ answers with regards the self-assessment of their learning experience in the questions of the four dimensions.

Discussion

This case study presents the preliminary findings of an ongoing research work on effective STEAM learning practices. The above methodology and survey results stated that the blended learning astronomy/ STEAM program assessed via smart phones and working students’ engineering projects only with recycled materials can be effective in producing learning as triangulated from the two different sources. The program developed following the STEAM learning best practices: Project -based learning methodology (emphasizing on authenticity, multimodality, and interactivity), adaptation of the UDL principles, delivered in blended methodology. Acquiring astronomy “big ideas” (Retro et al., 2019) in the context of effective STEAM education can be an enriching and engaging experience providing participants with 21st Century Skills, STEAM Capacity and Interest. Results are in accordance with previous research work (National Research Council, 2009; Vrioni & Ioannou, 2021; Vrioni, Mavroudi & Ioannou; 2019, Yildirim, 2016; Kazu, & Kurtoglu Yalcin, 2021). Results indicate that astronomy/ STEAM program can be a future curriculum for ensuring effective astronomical literacy in both formal and informal settings, even in learning spaces (astro-parks, astro-camps and astro- villages) with limited infrastructure. Results of this research give indications that effective STEAM learning must not relate to high-expensive learning materials and infrastructure. Blended learning environment accessed via smart phones and other mobile apps used in an effective STEAM context can be very powerful learning tools. Results of this research though based on subjective assessment (based on students self-assessments and teachers evaluation base on observed behavior) provide another promising indication, in the light of the finding that traditional teaching and learning practices, are ineffective to cultivate scientific literacy and high order thinking skills (Organization for Economic Co-operation and Development-OECD, 2018 etc.). Results cannot be generalized. Future research plans include replicating research in a more scaled population during the pilot testing phase at “astrocamps”. Also include experimentation design with regards the effect of learning materials and infrastructure available to students in STEAM learning gains.








Table 4: Descriptive Statistics and Frequencies from Students’ answers with regards the self-assessment of their learning experience on the questions of the four dimensions.





The Self-efficacy Questionnaire Dimension



The Learning Effectiveness Dimension.



 




Q3: I understand learning activities expectations

Strongly Agree (N=27, 84.4%) 

Agree (N=5, 15.6%)



Q11: What could you say are the benefits from the astronomy/ STEAM learning experience





Sparking Interest (N=28, 87.5%)





Q4: I understand how my performance is evaluated

Strongly Agree (N=29, 90.6%) 

Agree (N=3, 9.4%)



Understanding STEAM Content and Knowledge (N=29, 90.6%)





Q5: Learning Activities effectively promote my learning

Strongly Agree (N=29, 90.6%) 

Agree (N=3, 9.4%)



Engaging in Scientific, Engineering and Design thinking Processes (N=28, 87.5%)





Q7: I am active participant in my learning

Strongly Agree (N=26, 81.3%) 

Agree (N=6, 18.8%)



Develop my 21st century skills (N=29, 90.6%)





Q9: I have opportunities to learn with and from other students 

Strongly Agree (N=26, 81.3%) 

Agree (N=6, 18.8%)



Learn about STEAM careers I can follow (N=27, 84.4%)





Other (N=27, 84.4%)





The Learning Motivation Dimension



The Learning Satisfaction Dimension. 





Q2: I feel challenged to learn

Strongly Agree (N=27, 84.4%) 

Agree (N=5, 15.6%)



Q1: I feel welcome and respected

Strongly Agree (N=20, 62.5%)

Agree (N=12, 37.5%)





Q6: Learning Activities effectively promote my interest on the field

Strongly Agree (N=26, 81.3%) 

Agree (N=6, 18.8%)



Q8: The feedback I receive on my work is useful for me in making changes and improvements

Strongly Agree (N=21, 65.6%)

Agree (N=11, 34.4%)





Q10: I know where to go for help in this course, if and when I need it

Strongly Agree (N=26, 81.3%) 

Agree (N=6, 18.8%)






References

 

Adams, J. P., & Slater, T. F. (2000). Astronomy in the national science education standards. Journal of Geoscience Education, 48(1), 39-45.

Audrey C. Rule, (2006). Editorial: The Components of Authentic Learning. State University of New York at Oswego, Journal of Authentic Learning, Volume 3, Number 1, Pages 1-10, August 2006.

Barrows, H. S. (1985). How to design a problem-based curriculum for the preclinical years (Vol. 8). Singapore, Asia: Springer Publishing Company.

Barrows, H. S. (2000). Problem-based learning applied to medical education. Springfield, IL: Southern Illinois University School of Medicine.

Bell, S. (2010). Project-based learning for the 21st century: Skills for the future. The Clearing House, 83(2), 39-43. doi: 10.1080/00098650903505415

Boud, D. (2000). Sustainable assessment: rethinking assessment for the learning society. Studies in Continuing Education, 22, 2, 151-167. DOI: 10.1080/713695728.

DiMartino, J., & Castaneda, A. (2007). Assessing applied skills. Educational Leadership, 64(7), 38.

Erdogan, N., Corlu, M. S., & Capraro, R. M. (2013). Defining innovation literacy: Do robotics programs help students develop innovation literacy skills?. International Online Journal of Educational Sciences, 5(1), 1-9.

Kazu, I. Y., & Kurtoglu Yalcin, C. (2021). The Effect of STEAM Education on Academic Performance: A Meta-Analysis Study. Turkish Online Journal of Educational Technology-TOJET, 20(4), 101-116.

Ke, F. (2014). An implementation of design-based learning through creating educational computer games: A case study on mathematics learning during design and computing. Computers & Education, 73, 26-39. 

Kolodner, J., Camp, P., Crismond, D., Fasse, B., Gray, J., Holbrook, J., ... & Ryan, M. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning by design (tm) into practice. The Journal of the Learning Sciences, 12(4), 495-547. 

Li, S., Huang, X., Zhao, L., & Liu, J. (2020). A review of research on AR/VR in education from 2005 to 2018. Educational Research Review, 30, 100326.

Liu, D., Zhang, D., & Wang, C. (2020). A review of research on VR/AR in education from 2005 to 2019. Educational Research Review, 30, 100326.

Makransky, G., Terkildsen, T. S., & Mayer, R. E. (2019). Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learning and Instruction, 60, 225-236.

National Research Council. (2009). Learning science in informal environments: People, places, and pursuits. National Academies Press.

Noweski, C., Scheer, A., Büttner, N., von Thienen, J., Erdmann, J., & Meinel, C. (2012). Towards a paradigm shift in education practice: Developing 21st century skills with design thinking. In Design thinking research (pp. 71-94). Berlin, Heidelberg: Springer- Verlag. 

Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & evaluation in higher education, 35(4), 435-448.

Retrê, J., Russo, P., Lee, H., Penteado, E., Salimpour, S., Fitzgerald, M., ... & Schrier, W. (2019). Big ideas in astronomy: A proposed definition of astronomy literacy. Online] https://www. iau. org/static/archives/announcements/pdf/ann19029a. pdf.

Ronis, D. L. (2008). Problem-based learning for math & science: Integrating inquiry and the internet. Corwin Press.

Roth, W. M., Robin, K., & Zimmermann, A. (2002). Coteaching/cogenerative dialoguing: Learning environments research as classroom praxis. Learning Environments Research, 5(1), 1-28.

Rotherham, A., & Willingham, D. (2009). 21st century. Educational leadership, 67(1), 16-21. Retrieved from http://cesa7ita2009.pbworks.com/f/21st+Century+Skills+Curriculum+Teachers+Assessment.pdf

Sawyer, R. (Ed.). (2005). The Cambridge handbook of the learning sciences. Cambridge, UK: Cambridge University Press.

Snape, P., & Fox-Turnbull, W. (2011). Twenty-first century learning and technology education nexus. Problems of Education in the 21st Century, 34, 149-161. Retrieved from http://www.scientiasocialis.lt/pec/files/pdf/vol34/149-161.Snape_Vol.34.pdf

Stahly, L., Krockover, G. and Shepardson, D. 1999. Third grade students’ ideas about the lunar phases. Journal of Research in Science Teaching, 36(2): 159–177.  

Stroud, N., Groome, M., Connolly, R., & Sheppard, K. (2007). Toward a methodology for informal astronomy education research. Astronomy Education Review, 5(2), 146-158.

Thomas, J. (2000). A review of research on project-based learning. Buck Institute for Education. Retrieved from https://documents.sd61.bc.ca/ANED/educationalResources/StudentSuccess/A_Review_o f_Research_on_Project_Based_Learning.pdf

Tomar, G., & Garg, V. (2020). Making STEAM Accessible for Inclusive Classroom. Global Journal of Enterprise Information SySTEAM, 12(4), 94-101.

Vrioni Andri, (2022). Educating tomorrow's inventors. «Discovering the universe and the world of astronomy. Integration of astronomy into STEAM. Educating the inventors of tomorrow, cultivating innovation and skills of the 21st century." STEAM interdisciplinary activities to diversify teaching and create an enriched learning environment. ISBN 978-9925-7629-5-8

Vrioni Andri, 2019a. Educating tomorrow's inventors. Cultivating 21st Century Innovation and Skills at Kindergarten. STEAM interdisciplinary activities to diversify teaching and create an enriched learning environment. ISBN 978-9925-7629-0-3

Vrioni Andri, 2019b. Educating tomorrow's inventors. Cultivating 21st Century Innovation and Skills at Elementary School. STEAM interdisciplinary activities to diversify teaching and create an enriched learning environment. ISBN 978-9925-7629-1-0

Vrioni Andri, 2019c. Educating tomorrow's inventors. Cultivating 21st Century Innovation and Skills at the Secondary Education. STEAM interdisciplinary activities to diversify teaching and create an enriched learning environment. ISBN 978-9925-7629-2-7 

Vrioni Andri, 2019d. Why Christmas is so STEAM finally! Cultivating 21st Century Innovation and Skills. STEAM interdisciplinary activities to diversify teaching and create an enriched learning environment. ISBN 978 9925 7629 3 4

Vrioni, A. & Ioannou, I (2021). Measuring STEAM Learning Outcomes with an Authentic Assessment Tool. In Proceedings of the Panhellenic and International Conference on STE(A)M Educations and Educator, (MCIS), Patra, Greek, May 2021.

Vrioni, A., Mavroudi, A., & Ioannou, I. (2019, October). Promoting Authentic Student Assessment for STEAM Project-Based Learning Activities. In Interactive Mobile Communication, Technologies, and Learning (pp. 117-126). Springer, Cham.

Wu, H.-K., Huang, Y.-M., Chiu, M.-H., & Chen, Y.-C. (2021). Review of research on virtual and augmented reality in education: An extended bibliometric analysis. Educational Research Review, 33, 100362.

Yildirim, B. (2016). An Analyses and Meta-Synthesis of Research on STEAM Education. Journal of Education and Practice, 7(34), 23-33.

Zhou, Y., Zhang, D., & Liu, D. (2019). A review of research on AR/VR/MR in education from 2007 to 2018. International Journal of Educational Research, 96, 1-14.