# Leveraging Large Language Models to Support Authoring Gamified Programming Exercises +
## 4 Evaluation and Discussion
### Results and Discussion  
The experiment results (Table 3) show that most generated exercises were ready to be used with no or minimum human effort to make them comply with the expectations (61.7%).  
Nevertheless, there were non-neglectable cases in which the code snippet had to be manually added, especially for high-difficulty-level exercises (28.3%).  
Cases of entirely useless automatic generations were uncommon (10.0%).  
Another point to focus on is that the generated exercise quality was related to the kind of programming language. Most of the provided C-language exercises were ready to be used, whereas more Python exercises needed the human creator's intervention.  
The improvement in the speed of the exercise generation and gamification layer creation is beyond doubt.  
Nevertheless, some discussion about using the FGPE and GAMAI in a real-world production environment is needed. As shown by the example given in this section, the text generated by the GPT service could be affected by inaccuracies, biases, and intellectual property issues.  
While checking each automatically generated exercise is a task that could be considered mandatory for the content creator, and tedious but not complicated to perform, the same cannot be said about the quality check verification of the world created for the gamification scenarios' automatic generation. Arguably, due to the strategy used by Open AI to train and reinforce their models, they are strongly influenced by popular culture, causing a potential issue regarding copyright management.  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Score** & **Evaluation** \\ \hline
0 & The exercise can be used as is \\ \hline
1 & The text is unclear, it needs rearrangement \\ \hline
2 & The text must be completely rewritten, but the exercise is formerly correctly generated \\ \hline
3 & The text is correct, but there are issues from a technical point of view \\ \hline
4 & The exercise, although correct in the text part, is too easy or too hard considering the intended difficulty level \\ \hline
5 & The exercise cannot be used as is; it must be dropped or completely rewritten; for \\ example, a code snippet is entirely missing \\ \hline \hline \end{tabular}
\end{table}
Table 2: The evaluation grid for AI-generated programming exercise texts.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Prompt** & **0** & **1** & **2** & **3** & **4** & **5** & **Score** \\ \hline
1 CCS & 5 & 4 & 0 & 1 & 0 & 0 & 7 \\ \hline
1 PCS & 4 & 0 & 0 & 4 & 2 & 0 & 20 \\ \hline
1 PES & 6 & 0 & 0 & 4 & 0 & 0 & 12 \\ \hline
1 PLW & 3 & 1 & 2 & 3 & 0 & 1 & 19 \\ \hline
3 JCE & 4 & 3 & 0 & 3 & 0 & 0 & 12 \\ \hline
3 JCS & 6 & 1 & 0 & 0 & 3 & 0 & 13 \\ \hline
\% & 40.0 & 21.67 & 3.33 & 25.0 & 8.33 & 1.67 & \\ \hline \hline \end{tabular}
\end{table}
Table 3: Human effort needed to ensure usability of created gamified exercises. Experimental results.  
For example, in Listing 5, there is a field named \"wizarding_schools\" that conveys the name of three different schools of wizardry in the fantasy world of Eldoria [39]. In the used prompt (Listing 2), no mention was made of this entity, while the result produced is conceptually and numerically similar to literature protected by intellectual property rights. Note that this particular problem has also been observed in other use cases of Open AI's models [40].
```
1{
2'world':{
3'name'':'Eldoria'',
4'description'':'Amagicalrealmwherewizardingschoolsarescatteredacrossmysticallandscapes,andyoungsorcereersengagingalsolqueststoproretheirmagicalprowess.''
5},
6'wizarding_schools'':[...],
7'quests'':[
8{
9'name'':'TheEnchantedAmulet'',
10'description'':'RetrievetheEnchantedAmulethiddenwithintheForbiddenForest,guardedbymysticalcreaturesandprotectedbyapowerfulenchantment.'',
11'exercise'':{''repository'':'world__',''kind'':''multiple_choice'',''level'':'_iteration__'}
12],
13...
```  
**Listing 5** Example of an AI-generated gamified exercise (2).