## 5 Second Evaluation
### Evaluation Instruments and Techniques  
To continue evaluating the serious game, two different evaluation methods have been used. The first one was quantitative, while the second one was qualitative. Thus, a mixed approach [31] was used in this evaluation as the quantitative data gathered through a questionnaire was complemented by the opinions gathered through the focus group method.  
Firstly, a questionnaire similar to the one used by teachers was completed by the students. However, as in this case, the students were testing the full game, new items were added to the questionnaire. This questionnaire asks about personal information and presents a list of statements with which students needed to (dis)agree using a Likert scale  
\begin{table}
\begin{tabular}{l c c} \hline \hline
**Item** & **M** & **SD** \\ \hline \multicolumn{3}{c}{**Evaluation of Scrum VR**} \\ \hline My overall opinion is positive & 4.17 & 0.94 \\ The general narrative is adequate & 4.67 & 0.89 \\ The provided explanations are adequate & 4.42 & 0.67 \\ The artistic design is adequate & 4.33 & 0.89 \\ The objects shown (e.g., furniture, posters, blackboard, etc.) are adequate & 4.25 & 0.97 \\ The method of locomotion (i.e., how you move) is adequate & 3.83 & 0.84 \\ The accompanying sound is adequate & 4.58 & 0.52 \\ The stage lighting is adequate & 4.75 & 0.45 \\ The mechanisms of interaction are adequate & 3.58 & 1.17 \\ I have felt present in the experience (i.e., surrounded by the virtual world) & 3.75 & 0.97 \\ \hline \multicolumn{3}{c}{**Potential Academic Impact of Scrum VR**} \\ \hline It can promote learning about Scrum & 4.27 & 0.94 \\ It can help to understand a Scrum professional context & 4.17 & 0.72 \\ It can be engaging and motivating for learners & 4.5 & 0.52 \\ It can help to energize the teaching-learning process & 4.33 & 0.65 \\ \hline \multicolumn{3}{c}{**Evaluation of This Type of VR Application**} \\ \hline My general opinion about this type of application is positive & 4.45 & 0.93 \\ This type of application is innovative & 4.42 & 1.24 \\ This type of application is a good complement to traditional learning methodologies & 4.5 & 1.0 \\ I would like to use this type of application with my students & 4.08 & 1.38 \\ It would be useful to incorporate more applications like this in the University & 4.42 & 1.24 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Quantitative results: Teacher’s evaluation.  
from 1 (totally disagree) to 5 (totally agree), questions to select an answer from several options and open questions to gather comments about the aspects of the serious game they liked the most and the least. The items of the questionnaire are presented together with the results.  
Secondly, several focus groups were conducted to collect qualitative data from the students about the usage of the application in VR. The focus group method started to be used in social sciences and currently it is widely applied in many domains [32], including Software Engineering [33]. Focus groups are defined as a discussion involving 3 to 12 people and guided by a moderator who follows a predefined script to keep the discussion focused [33]. In our case, three focus groups were carried out, and each of them lasted approximately 25 min. The following topics were explored from a qualitative perspective:  
* General appreciation. What do you think about the application?
* Knowledge. What concepts about Scrum do you think are the most and least clear?
* Usefulness. How useful do you think it will be for the students who use it?
* Improvement. What aspects of the application do you think could be improved?
* Striking features. What aspects of the application have caught your attention the most?