Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses
Authors: Aysa Xuemo Fan
Haoran Ranran Zhang
Luc Paquette
Rui Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Model
Natural Language Processing
Computer Science Education
Novice Programming

Abstract: In this paper, we explore the application of large language models (LLMs) for generating code-tracing questions in introductory programming courses. We designed targeted prompts for GPT4, guiding it to generate code-tracing questions based on code snippets and descriptions. We established a set of human evaluation metrics to assess the quality of questions produced by the model compared to those created by human experts. Our analysis provides insights into the capabilities and potential of LLMs in generating diverse code-tracing questions. Additionally, we present a unique dataset of human and LLM-generated tracing questions, serving as a valuable resource for both the education and NLP research communities. This work contributes to the ongoing dialogue on the potential uses of LLMs in educational settings.

Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses
Aysa Xuemo Fan', Ranran Haoran Zhang", Luc Paquette', Rui Zhang?
University of Illinois at Urbana-Champaign
2 Penn State University
(xuemof2, 1paq) @ willl illliiis,,.. edu
(hz25361, rmz 5227 @psu. edu
Abstract
Cullected
Description:
Ideniity reverse
this paper, we explore the application of large language models (LLMs) for generating code-tracing questions in introductory program- ming courses. We designed targeted prompts for GPT4, guiding it generate code-trace code-tracing questions based on code snippets and descrip- tions. We established a set of human evalua- tion metrics to assess the quality of questions produced by the model compared to those cre- ated by human experts. Our analysis provides insights into the capabilities and potential of L.MS generating diverse code-tracing ques- tions. Additionally, we present a unique dataset of human and LLM- -eggerretd tracing ques- tions, serving as a valuable resource for both the education and NLP research communities. This work contributes to the ongoing dialogue on the potential uses of LLMs in educational settings'.
digits on credit card.
salppet:
1nt 365;
inz dun - 0F
whilee 03
digit 1C; wum aun digits = a/10;
I.I. M-gonerated Tracing Questions: the given code. the ourput if the value of
Systton out. pet.priniintemm
the requirement was to identify the reverse digits on eredit card and then return them a single integer, would you modify the code to accomplish this?
Hammnn-created Trucing
Questions:
What primtig?
Are the qualiiies quetions comparable?
Which tracing questions better?
Evaluation
Can experts distinguish between the authors (LLM or human)?
Who creates more diverse questions?
Figure We aim to assess the Large Language Models' (LLMs') capability to generate code tracing questions, pivotal in computer science education. The accompa- nying illustration outlines our approach, where GPT4 is employed to generate questions based given code snippets and descriptions. Subsequent comparative anal- ysis with human-created questions aids in exploring critical aspects, such as the quality and diversity of gen- erated questions, discernibility between human and authors, and the relative superiority in question quality.
1 Introduction and Background
The teaching of introductory programming courses continues to be challenging endeavor, despite the global uptake and popularity of such courses. High enrollment rates often result in diverse student pop- ulations, with a wide range of programming experi- ence from those just starting their journey to others with prior exposure (Lopez et al., 2008). Ensuring an effective learning experience that accommodates this wide disparity presents daunting task, making the teaching of these courses complex.
2018), and tracing quizzes (Sekiya and Yamaguchi, 2013). These strategies consistently utilize code- tracing questions aimed at fostering and developing a student's 's understanding and skills.
One critical component in teaching introductory programming the focus on code tracing, a skill identified as instrumental in enhancing code writ- ing abilities (Lister et al., 2009; Venables et al., 2009; Kumar, 2013). Current educational method- ologies encourage code tracing through variety of means, such as practice questionnaires (Lehtinen et al., 2023), direct teaching strategies (Xie et al.,
However, the preparation of code-tracing ques- tions poses challenges. Manual question creation by instructors (Sekiya and Yamaguchi, 2013; Has- and Zilles, 2021) is time-consuming and lacks scalability. Automatic generation using program analysis saves time, yet is limited by the analyzer's capabilities and lacks question diversity (Zavala and Mendoza, 2018; Thomas et al., 2019; Russell, 2021; Lehtinen et al., 2021; Stankov et al., 2023).
OOur data and code are available at https: //gltnub.com/aasaa com/aysaaanmmm/ i code_ loade_tracing tracing. ing_ inrestion_generatiio

---

In light of the increasing potential of Large Lan- guage Models (LLMs) in sectors like code summa- rization and explanation (Chen et al., 2021; Siddiq et al., 2023), the question arises: Can LLMs gener- ate high-quality code-tracing questions? Our study explores this query using GPT4 (OpenAI, 2023), leveraging prompts to guide its question generation based on given code snippets and descriptions. To assess the LLM'' capability in this pivotal aspect of computer science education, we devised set of hu- man evaluation metries. This allowed for an ohjec- tive appraisal the LLM- -LM-generated questions, and, through a comparative analysis with cumann-reeated counterparts, critical aspects such as question qual- ity, diversity, discernibility between human and AT authors, and relative superiority in quality were ex- plored (Figure 1). These analyses have enhanced our understanding of the potential roles of LLMs in computer science education.
complex tasks like generating unanswerable ques- tions (Choi et al., 2018: Zhu et al., 2019; Reddy et al., 2019) and multi-hop reasoning (Pan et al., 2020, 2021; Shridhar et al., 2022). Our work, focus- ing on generating code tracing questions CS Edu- cation domain, addresses unique challenges around code, natural language. and pedagogical compre- hension, inadequately covered by previous methods due lack of specialized datasets.
Code LLMs for CS Education: Recent ad- vances in code large language models (LLMs) (Chen et al., 2021; Wang et al., 2021: Le et al.. 2022; Wang et al., 2023) have enabled various downstream applications, including code comple- tion, retrieval, summarization, explanation, and unit test generation (Lu et al.; Siddiq et al., 2023; Tian et al., 2023). Studies have showcased the LLMs' ability to generate novice programming content comparable to humans (Finnie-Ansley et al., 2022; Piccolo al., 2023). L.LMs have been utilized classroom environments (Kazemitabaar et al., 2023), to generate coding exercises and ex- planations (Sarsa et al., 2022), and to create coun- terfactual questions (Narayanan et al., 2023). Our study represents the first exploration of LLMs for code tracing question generation, critical compo- nent of CS Education, thus underscoring the po- tential of these models for generating educational content.
This investigation provides a foundation for con- sidering the potential inclusion of LLMs in learning platforms, which could offer new possibilities for enhancing the learning experience in introductory programming courses. Given these advancements, our study contributes to the field follows:
The curation of high-quality dataset consist- ing of human and LLM-generated code trac- ing questions and associated code snippets.
An exploration and evaluation of GPT4's ca- pability question generation, including com- parisons with both GPT3.5-turbo and human- authored questions, and an examination of
Our Approach
3.1 Task Definition
few-shot and zero-shot scenarios.
In automatic tracing question generation, given description (optional) d E DUØ, detailing the code context, and code snippet E C provided by an instructor or student, the aim to generate set of relevant questions Q' for student practice. This task can be formally defined as a function:
The introduction of human evaluation methodology and comprehensive assess- ment of the quality of L..M-generated ques- tions, offering valuable insights into the po- tential of LLMs in educational contexts.
f
(1)
2 Related Work
Early Question Genera- tion (QG) research primarily focused on multiple- choice questions (Mitkov et al., 2006; Agarwal and Mannem, 2011) and questions with specific inter- rogatives (Heilman and Smith, 2010). With the emergence of the SQuAD dataset (Rajpurkar et al., 2016), context-dependent QG gained momentum (Du et al., 2017; Yuan et al., 2017; Subramanian et al., 2018; Puri et al., 2020). This extended to
Question Generation:
where D represents all possible descriptions, C' all possible code snippets, and Q' is subset of all possible questions Q.
3.2 Curating the Code- Code-Tracing Question Dataset
For our experiment, we curated a unique dataset re- flecting the range of tracing questions encountered by beginner programmers. We sourced 158 unique

---

we employ regular expressions in a postprocess- ing step segment the generated content, isolat- ing individual tracing questions. Subsequently, for each generated prediction p. its BERTScore is computed in relation to the reference, denoted as BERTScore(refcrenee p).
BERT Scores Comparison
0.8
Score
The boxplot Figure 2 displays the Precision, Recall, and Fl scores for both models. From the graph, it's clear that GPT-3.5-turt has a median Precision score around 0,45, Recall slightly above 0.6, and an F1 score hovering around 0.5. In comparison, GPT-4 shows more balanced perfor- mance with median Precision score close to 0.6, Recall near 0.55, and F1 just above 0.5. Notably, the variability in scores, particularly for GPT-4, highlights the diverse outcomes in its results. Based on our results, we chose GPT4 for subsequent eval- uations. Enhanced performance examples from GPT4 are in Appendix C.
onectsion
-0.2
GPT-3.5
GPT-4
Model
Figure 2: Comparison the BERT on all LLM- generated questions and human-authored questions.
questions from CSAwesome', recognized online Java course aligned with the AP Computer Science A curriculum. To enhance diversity, we added 18 questions extracted from relevant YouTube videos. Other platforms and sources were also examined but didn't fit due to a lack of explicit tracing ques- tions. Our final dataset consists of 176 unique code snippets and question pairs, allowing a fair eval- uation of LLMs' ability to generate introductory programming tracing questions.
Next, we hypothesized that the few-shot ques- tion generation approach, which feeds the model with three tracing question examples and their re- spective code snippets, would yield higher-quality questions than the zero-shot generation that relies solely on the prompt. Contrary to our expectations, the experiment showed that the few-shot method introduced significant bias towards the example questions, thus narrowing the diversity in the gen- erated questions. Consequently, we opted for the zero-shot generation in our tests, which fostered a broader spectrum of question types. Detailed ex- amples of outcomes from both the zero-shot and few-shot approaches are available in Section 4.4.
3.3 Prompt Engineering and Model Selection our iterative approach to prompt engineering and model selection, we first refined prompts and then generated tracing questions using GPT-3.5- turbo and GPT-4. Using BERTScore, we assessed question diversity and similarity. Based on these insights, we combined the optimized prompt with the chosen model to determine the most effective generation approach, be it few-shot or zero-shot. Our final prompt, refined iteratively from (Brown al., 2020), positioned in Appendix B, adopts an expert instructor's perspective, encourages deep understanding via code-tracing questions, and max- imizes the inherent versatility of LLMs.
3.4 Human Evaluation
Next, we conducted human evaluation compar- ing the quality of GPT4-generated and human- authored tracing questions. The expert evaluators were meticulously screened based on specific cri- teria: they had to be computer science graduate students with at least one year of programming teaching or tutoring experience. Four such experts, meeting these criteria, participated in the evalua- tion.
Next, we considered GPT-3.5-turbo and GPT-4 for model selection, and investigate the generated tracing questions diversity by BERTScore (Zhang* al., 2020). Regarding the automatic evaluation of the diversity in generated questions, we adopted the following methodology: For each code snip- pet, we utilize a singular human-authored tracing question as the reference. Both GPT3.5-turbo and GPT4 are then tasked with generating multiple trac- ing questions for every snippet. Following this,
Each evaluator was assigned a set of 44 ran- domly selected code snippets from pool of 176 human-authored tracing questions. For each snip- pet, evaluators received a pair of questions (one human-authored and one GPT4-generated) in a ran- domized order to mitigate potential ordering bias. Evaluators unawareness of question authorship was
'nttos://runeste academy // bbokk// mublisshed/caaweeee index. html

---

Criteria
Label
teria in Table 3. Significant differences emerged in three criteria: relevance learning objectives, clarity, and difficulty appropriateness. However, the relevance to the given code snipper showed no significant difference, indicating comparable per- formances.
Relevance Learning Objectives Tracing or Clarity of the Question
1-5
Yes/No
1-5
Difficulty Level
1-5
Relevance the Given Code Snippet Ability Distinguish Source
1-5
Human-createdd Al-generated Check preferred
Despite U-tests highlighting significant differ- ences in some criteria, the practical quality differ- ence was minimal. As further detailed in Table 2. LLM-generated questions had slightly lower mean ratings, yet their median ratings closely mirrored those of numan-authored questions.
Preference for Better Question
Table 1: Criteria used for expert evaluation.
Human
GPT4
Mean
Modian
Mean
Median
Considering these two analyses together, is apparent that despite some statistical differences, LLM-ggnnrrted questions still maintain high ped- agogical standard effectively. Consequently. LLM, while underlining areas for potential enhancement, demonstrates proficiency in generating questions that align closely in quality and course relevance with those crafted by humans.
Relevance to Learning
4.78
5.00
4.62
5.00
Question Clarity
4.72
5,00
4.42
5,00
Appropriate Difficulty
4.75
5.00
4.43
5.00
Relevance to Code
4.72
5,00
4.64
5,00
Table 2: Comparative statistics for human and GPT4 generated questions.
ensured.
The evaluators rated each question based on five criteria shown in Table 1. They also guessed the question's 's authorship and expressed their prefer- ence between the pair. Detailed evaluation criteria and labels can be found in Table 1.
4.2 Expert Perception of Question Authorship
We further evaluated the discernibility of LLM- generated questions from human-authored ones using Confusion Matrix (Table 5). Approxi- mately 56% (99 out of 176) of GPT4-generated questions were mistakenly identified by experts as human-generated, and about 20% (35 out of 176) of human-authored questions were misattributed to the GPT4. This overlap signifies the high quality of the generated questions and GPT4's proficiency in producing pedagogically relevant tracing ques- tions. Moreover, the matrix reveals an evaluator bias toward attributing higher-quality questions human authorship.
4 Analyses and Results
This section details analysis and highlights the results, encompassing quality ratings, expert per- ceptions, and textual similarities question gener- ation.
4.1 Comparative Analysis of Quality Ratings
U-val
Relevance to learning objectives
3688.0
0.047
4.3
Textual Similarity between Questions
Question Clarity
3392.0
0.011
Difficulty Appropriateness
3540.5
0.015
Table 6 presents BLEU (Post, 2018), ROUGE- 1/2/L (Lin. 2004), and BERTScores (Zhang al., 2020), comparing the similarity between the ran- domly selected GPT4 generated questions and cor- responding human-authored questions. The low BLEU and ROUGE scores suggest that GPT4 is generating distinct, non-verbatim questions com- pared to numan-authored questions. A moderate BERTScore, reflecting semantic similarity, sug- gests that GPT4-generated questions align with the context of numan-authoree ones. This further underscores GPT4's capability to independently generate relevant and diverse code-trad code-tracing ques- tions, distinct from those crafted by humans.
Relevance the given code snippet
3918.5
0.595
Table 3: Results of Mann- Whitney tests compar- ing human evaluations of LLM-ggnerated and human- authored code-tracing questions. U-values and p-values are provided for four criteria. Significant differences for three criteria indicate varying performance, while no difference in 'relevance to the given code snippet' suggests similar performance.
To assess the quality disparity between LLM- generated and human-authored questions, we ap- plied Mann- Mann-Whitney U tests (Mann and Whitney, 1947) to the median ratings of four evaluation cri-

---

GPT-4: Few-Shot vs. Zero-Shot Results
Zero-Shot Prompt:
Few-Shot Prompt:
Your task Craft tracing questions for: (requirement) (code snippet)
Your task [oode_snippet]] (trucing questionl| for: |requirement) {code_snn ]
Examples:
(requirement] )
Craft
Zero-Shot Result:
Few-Shot Result:
1. What is the purpose of this function? What does it
Tracing question I: What is the output of the method redo(9, 3)?
aim to accomplish?
2. What is the base case for this recursive function?
Tracing question 2: What is the output of the method redo(O, 5)?
What happens when the base
reached? 3. If the input values are = 88 and what will be
Tracing question 3: What is the output of the method redo(27, 3)?
the output of the function? Please trace through the code step-by-step and explain your reasoning.
6. What happens if both input values negative, such i= -8 and -2? Does the function handle this case correctly according the requirement? Explain your reasoning.
Table 4: Illustrative comparison between GPT-4's 's code-tracing question generation in Few-Shot and Zero-Shot settings, showcasing the diversity and specificity of generated questions.
Predicted GPT4
Predicted Human
Metric
Score
BLEU
0.02
Actual GPT4 Actual Human
77
99
ROUGE-1 F-score
0.215
35
141
ROUGE-2 F-score
0.051
ROUGE-L F-score
0.199
Table 5: Confusion Matrix indicating experts' attribu- tions of code-tracing questions. The table displays the number of actual GPT-4 generated and human- authore questions and how they were predicted by the experts, underscoring the challenge distinguishing between the two.
BERTScore Precision
0.274
BERTScore Recall
0.341
BERTScore F1
0.303
Table 6: Comparing the similarity between human- authored questions and the random choiced GPT4 ques- tion.
Combining with the previous analyses, the LLM, such GPT4, thus exhibits substantial promise in generating high-quality, course-relevant code- tracing questions, illustrating its utility as teach- ing aid.
the influence of training data and model design in zero-shot scenarios, allowing GPT-4 to tap into its vast training experience. contrast, in few-shot scenarios, the model might overly adhere to the provided examples, interpreting them as stringent templates, which can compromise output diversity.
4.4 Few-shot vs Zero-shot Generation Results
The balance between the nature of the task and the examples becomes pivotal in few-shot settings, potentially leading outputs that may sacrifice accuracy or diversity. These hypotheses warrant further investigation future work.
Few-shot generation biased our model towards the provided examples, largely reducing question di- versity. In contrast, zero-shot generation yielded more diverse questions, prompting us to favor for broader question variety in our experiment. De- tailed examples of the generated results for both O-shot and few-shot methods can be found in Ap- pendix D.
Conclusion
This study explored the capability of GPT-4 in gen- erating code-tracing questions that rival the quality of those crafted by human educators. The findings illuminate the potential of LLMs to bolster pro- gramming education, marking a significant stride in the domain of code-tracing question generation and LLM application. This sheds light on scalable, high-quality automated question generation.
Table provides a bideeby-side comparison of GPT-4's performance in few-shot and zero-shot set- tings. The zero-shot results exhibit a broader range of question types, while the few-shot results seem to be more templated, reflecting the bias introduced by the provided examples.
Possible reasons for these observations include

---

Limitations and Future Work
within the educational ecosystem vital. Our fu- ture work will be guided by these ethical principles of privacy, informed consent, secure data handling. inclusivity, and conscientious progress focused on students' hest interests.
This study marks a step closer in evaluating LLMs for code tracing question generation, but it is not without its limitations. Our research was primarily anchored to GPT-4, raising concerns about the gen- eralizabillity of our findings to other LLMs, such as CodeT5+. Moreover, the study did not delve into the personalization of tracing questions based on individual student submissions, a facet that could greatly enhance the learning experience. Further- more, the real-world educational efficacy of the LLM-generated questions remains an open ques- tion, given that our study did not involve actual students.
References
Manish Agarwal and Prashanth Mannem. 2011. I. Auto- matic gap-fill question generation from text books. In Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications. pages 56 64, Portland, Oregon. Association for Com- putational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Nlaa Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates. Inc.
Several avenues beckon for further exploration. Evaluations with a broader range of models will offer a more comprehensive perspective on L.M capabilities. While our study centered on intro- ductory Java tracing questions, assessing LLM ver- satility across different programming domains is imperative. The potential of LLMs exiends beyond mere question generation; by tailoring questions to student needs, we can amplify the educational relevance. Our roadmap includes the development of an educational platform integrated with LLM questions, followed by classroom experiments and usability testing. To ensure broader applicability, expanding our dataset is crucial. Lastly, our find- ings on few-shot and zero-shot learning necessitate further investigation into model adaptability, bi- ases in question generation, and the potential of intermediate-stoe shot learning.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser. Mohammad Bavarian. Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Tvvan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained code.
These directions not only underscore the trans- formative potential of LLMs AT-driven education but also emphasize the importance of comprehen- sive evaluations.
Ethical Statement
Our exploration of Large Language Models (LLMs) in introductory programming education was conducted ethically. We sourced public data and maintained evaluator anonymity and data confi- dentiality through secure storage. Evaluators were informed of the objectives and participated volun- tarily. All evaluation results, as committed in the IRB forms, are securely stored. We strived for ed- ucational faimess by properly compensating the educators involved in our evaluation. We are mind- ful of the societal impacts of LLM integration in education. While acknowledging their promise, we believe careful consideration of pedagogical goals
Eunsol Choi, He He, Mohit lyyer. Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018 QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2174-2184, Brussels, Belgium. Association for Com- putational Linguistics.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn ing to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual

---

Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), pages 1342-1355 1352, Vancouver, Canada. Association for Computational Linguistics.
Raymond Lister, Colin Fidge, and Donna Teague. 2009. Further evidence of relationship between explain- ing. tracing and writing skills in introductory pro- gramming. Acm sigese bulletin, 41(3):611-665.
James Finnie-Ansley, Paul Denny, Brett A. Becker, An- drew Lutoo--Reilly and James Prather. 2022. The
Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. 2008. Relationships between read- ing, tracing and writing skills in introductory pro- gramming. In Proceedings of the fourth international workshop on computing education research, pages 101-112.
robots are coming: Exploring the implications of ope- nai codex on introductory programming. In Proceed- ings of the 24th Australasian Computing Education Conference, ACE '22. page 10-19, New York, NY, USA. Association for Computing Machinery.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. Thirty- fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 1).
Mohammed Hassan and Craig Zilles. 2021. Exploring 'reverse-tracine questions as means of assessing exams. In
the tracing skill on computer-based Proceedings of the 17th ACM conference on interna- tional computing education research, pages 115-126.
Michael Heilman and Noah A. Smith. 2010. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 An- nual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609-617, Los Angeles, California. Association for Computational Linguistics.
Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochasti- cally larger than the other. The annals of mathemati- cal statistics, pages 50-60.
Ruslan Mitkov, Ha Le An, and Nikiforos Karamanis. 2006. A computer-aided environment for generating multiple-choice test items. Natural language engi- neering, 1772):177-194.
Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma. Bar- bara J. Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of ai code generators on supporting novice learners in introductory program- ming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, New York, NY, USA. Association for Computing Machinery.
Arun Balajiee Lekshmi Narayanan, Rully Agus Hen- drawan, et al. 2023. Enhancing programming etextbooks with chatgpt generated counterfactual- thinking-inspired questions. arXiv preprint arXiv.23060005//.
Amruth N Kumar. 2013. A study of the influence of code-tracing problems on code- code-writing skills. In Pro- ceedings of the 18th ACM conference on Innovation and technology in computer science education, pages 183-188.
OpenAL. 2023. Gpt-4 technical report.
Liangming Pan, Wenhu Chen, Wenhan Xiong, Min- Kan, and William Yang Wang. 2021. Unsupervised multi-hop question answering by question generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5866-588 Online. Association for Computa- tional Linguistics.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. 2022. Coderl: Mas- tering code generation through pretrained models and deep reinforcement learning. In NeurIPS.
Teemu Lehtinen, Lassi Haaranen, and Juho Leinonen. 2023. Automated questionnaires about students' javascript programs: Towards gauging novice pro- gramming processes. In Proceedings of the 25th Aus- tralasian Computing Education Conference, pages 49-58.
Liangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua, and Min-Yon Kan. 2020. Semantic graphs for generating deep questions. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguisties, pages 1463-1475, Online. Asso- ciation for Computational Linguistics.
Teemu Lehtinen, André L Santos, and Juha Sorva. 2021. Let's ask students about their programs, automati- cally. In 2021 IEEF/ACM 29th International Con- ference on Program Comprehension (ICPC), pages 467-475. IEEE
Stephen R Piccolo, Paul Denny, Andrew Luxton-Reilly, Reilly.
Samuel Payne, and Perry G Ridge. 2023. Many bioin- formatics programming tasks can be automated with chatgpt. arXiv preprint arXiv:2303. 13528.
Matt Post. 2018 A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.

---

Raul Puri. Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. 2020. Training question answering models from synthetic data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3811-5826, Online. Association for Computa- tional Linguisties.
and question generation. In Proceedings of the Work- shop on Machine Reading for Question Answering. pages 78-88, Melbourne, Australia, Association for Computational Linguistics.
Anderson Thomas, Troy Stopera, Pablo Trank-Bolton and Rabul Simha. 2019. Stochastic tree-based gen- eration of program -aranin practice questions. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education, pages 91-97.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016, SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing- Chi Cheung, Jacques Klein, and Tegawendé F Bis- syandé. 2023. Is chatgpt the ultimate program- ming assistant-how far is it? arXiv preprint arXiv:2304.1 1998.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-266 266.
Anne Venables, Grace Tan, and Raymond Lister. 2009, A closer look at tracing, explaining and code writing skills in the novice programmer. In Proceedings of the fifth international workshop on Computing education research workshop, pages 117-128.
Scán Russell. 2021.
Automatically generated and graded program tracing quizzes with feedback. In Proceedings of the 26th ACM Conference on Innova- tion and Technology in Computer Science Education V. 2, ITiCSE '21, page 652, New York, NY, USA. Association for Computing Machinery.
Yue Wang, Hung Lc. Akhilesh Deepak Gotmare, Nghi D.Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. Codet5+: Open code large language mod- els for code understanding and generation. arXiv preprint.
Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic generation program- ming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research- Volume 1, ICER '22, page 27-43, New York. NY, USA. Association for Computing Machin- ery.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeTT: Identifier-aware unified pre- trained encoder-decoder models for code understand- ing and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8696-8708, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Takayuki Sckiya and Kazunori Yamaguchi. 2013. Trac- ing quiz set to identify novices' programming mis- conceptions. In Proceedings of the 13th Koli Calling International Conference on Computing Education Research, pages 87-95.
Benjamin Xie, Greg L Nelson, and Amy J Ko. 2018. An explicit strategy to scaffold novice program tracing. In Proceedings of the 49th ACM technical symposium on computer science education, pages 944-349.
Kumar Shridhar, Jakub Macina, Mennatallah El-A El-Assady, Tanmay Sinha, Manu Kapur, and Mrinmaya Sachan. 2022. Automatic generation of socratic subquestions for teaching math word prohlems. Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 4136-4149, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Xingdi Yuan, Tong Wang, Caglar Gulcehre, Alessandro Sordoni, Philip Bachman, Saizheng Zhang, Sandeep Subramanian, and Adam Trischler. 2017. Machine comprehension by text-to-text neural question gener- ation. In Proceedings of the 2nd Workshop on Repre- sentation Learning for NI.P, NaPep 15-25, Vancouver, Canada. Association for Computational Linguistics.
Mohammed Latif Siddiq, Joanna Santos, Rid- wanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes 2023. Exploring the effectiveness of large language models in generating unit tests. Xixiv preprint arXiv:2305.004 00478.
Laura Zavala and Benito Mendoza. 2018 On the use of semantic- based aig to automatically generate pro- gramming exercises. In Proceedings of the 49th ACM Technical Symposium on Computer Science Educa- tion, SIGCSE '18, page 14-19, New York, NY, USA. Association for Computing Machinery.
Emil Stankov, Mile Jovanov, and Ana Madevska Bog- danova. 2023. Smart generation of code tracing ques- tions for assessment in introductory programming. Computer Applications in Engineering Education, 31(1):5-25.
Tianyi Zhang R Varsha * Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Sandeep Subramanian, Tong Wang, Xingdi Yuan, Saizheng Zhang, Adam Trischler, and Yoshua Ben- gio. 2018. Neural models for key phrase extraction
Haichao Zhu, Li Dong, Furu Wei, Wenhui Wang, Bing Qin, and Ting Liu. 2019. Learning to ask unanswer- able questions for machine reading comprehension.

---

In Proceedings of the 57th Annual Meeting the As- sociation for Computational Linguistics. pages 4238- 4248, Florence, Italy, Association for Computational Linguistics.

---

A Questionare
Description: (blank)
Code:
int [10 []
(2,_ 12,46,6)))
int sum 0;
for (int 0; k m.lenghh; k++)
sum sum m mmm a.m.lenghh--11 [1];
Question 1: Given the following code segment, what is the value of sum after this code exccutes? Question 2: What is the role of the expression im/m.length-1-k[11 in the code? Annotation
1. Relevance Learning Objectives: The question is relevant to the learning objectives of an introductory programming course. (Label: 1-5)
Tracing or not: Is this tracing question? (Label: Yes or No)
3. Clarity of the Question: The question presented is clear and the language used in the question is easy to understand. (Label: 1-5)
4. Difficulty Level: The difficulty level of the question is appropriate for an introductory programming
course. (Label: 1-5)
5. Relevance to the Given Code Snippet: The question is appropriately related the code snippet
provided in the question. (Label: 1-5)
Ability to Distinguish fluman-Authored from Automatically Generated Questions: Can you tell if the question is numan-authoreed or automatically generated? (Label: Human-created or Al-generated) think this is a better tracing question. (Check the box under the better question)
This an example of our questionnaire sent annotators.
B Prompts
The final prompt we employed offers LLMs detailed context: requests the generation of questions from an expert instructor's 's perspective within a defined pedagogical setting. It outlines the merits of code-tracing questions, gives an insight into their typical structure, and highlights their educational aim, specifically, promoting in-depth understanding rather than just assessing knowledge. Unlike the data collection process, where each code snippet linked single question, the prompt is designed produce multiple valid tracing questions for the same snippet, which leverages the inherent diversity and breadth potential of LLMs.

---

User Prompt: In your role education expert in introductory Java programming course, you are equipped with a deep understanding of Java and teaching methodologies. Your aim shape the minds of young learners, paving their path to becoming proficient programmers. One of your potent teaching techniques involves providing students with real-world coding practice requirements and letting them craft working code snippets.
Consider a scenario where you have presented your students with a requirement to develop a basic calculator program in Java, performing fundamental arithmetic operations like addition, subtraction, multiplication, and division. Enthusiastically, the students pour their effort into crafting varied solutions, reflecting their unique approach to the problem and their understanding of Java programming.
But your task doesn't t end with the generation of code. It's now time for the students delve deeper, and here comes the importance of tracing programming. Code tracing is the process of manually going through the execution of code, step-by-step. to understand its flow and logic. It's a fundamental practice in learning programming, aiding students in debugging their programs and developing a stronger understanding of code behavior.
One of the best ways to encourage code tracing is by generating tracing questions. Such questions stimulate the students to analyze their code at granular level, fostering their problem-solving skills and enhancing their understanding of the Java language.
Remember, your questions should stem from two sources the initial Requirement and the Code the students have produced. For example, given code snippet, you might ask 'What is the output of this code if the user inputs two and three and chooses multiply?" or 'Can this code handle the division by zero scenario, according to the requirement?"
Your goal is not just to evaluate their coding skills, but to instill a deep, lasting understanding of Java programming in them. By asking relevant tracing questions, you're re helping to nurture new generation of critical thinkers, problem solvers, and proficient programmers.
You will ask many relevant tracing questions given 1. Requirement, 2. Code, below: Requirement: (blank) Code:
int 3;
int y 2;
if (x 2)
x++;
if (y 1)
y++;
if (x > 2)
System. wwt.outt.iii... (efirst ("first_")
if (y 3)
System. out. print ("second_")
System. out, print ("third");
This is an example of our prompt.
C GPT3.5 vs GPT4
Prompt... Requirement: (blank) Code:

---

int 3;
int 2;
if (x 2)
x++;
(y 1)
y++i
if (x 2)
System. out. print ("first_")
if (y 3)
System. out. print ("second_") ; System. out print ("third");
GPT-3.55 Result:
Tracing Questions:
1. What are the initial values of and y in this code?
2. Does the if statement line 3 execute? Why why not?
3. Does the if statement line execute? Why why not?
What is the value of after line has executed?
What is the value of y after line has executed?
Which cestem.out.prie statement(s) will execute this code?
What is the output of this code?
8. What happens if the initial value of x is 1? How does the output change?
9. What happens if the initial value of is 3? How does the output change?
10. Can this code handle other arithmetic operations besides addition and subtraction, according to the requirement?
GPT-4 Result: 1. What will be the values of x and y after executing the first two if statements?
2. What is the output of the code snippet it is?
If we change the initial value of 1, what will be the output of the code snippet? If we change the initial value of O, what will be the output of the code snippet?
What will happen both x and y are initialized with values less than or equal to 2? What will be the output in that case?
6. How would you modify the code to include an additional condition where it checks if x is equal to and prints "equal" if it's true?
7. If we swap the positions of the second and third if statements, would the output change? Why or why not? 8. Can you explain the flow of the code exccution given the initial values of and y? What the key decision points in the code?
What happens if we change the second if statement "if (y 1)"? How does that change the output of the code snippet?
10. How would you modify the code to print "none" none the conditions are met in the statements?
This example of GPT3.5-turbo v.s. GPT4 for tracing question generation on a simple java problem.
Prompt Requirement: (blank) Code:

---

Requirement: (blank)
Code:
public static int redo (int i, int j)
if (i==0)
return 0;
else
return redo (i/i, +)
GPT-3.55 Result:
1. What is the purpose of the redo() method?
2. What is the return type of the redo() method?
3. What is the initial value of and j when the redo() method is called?
4. What happens if the value of is 0?
5. What happens if the value of j is 0?
6. What happens if the value of is negative?
What happens if the value of is negative?
What happens if the value of is greater than j?
9. What is the output of the redo() method the values of and are 8 and 2 respectively?
10. What is the output of the redo() method if the values and are 10 and 3 respectively?
GPT-4 Result:
1. What is the base case for this recursive function, and what does it return?
2. How is the recursion progressing towards the base case in this code?
3. What will be the output of this function if the inputs are i=16 and j=2?
4. What will be the output of this function if the inputs are i=81 and j=3?
5. How will the function behave if both i and equal to 0? Will it cause any issues?
6. How will the function behave if is less than j? Is the output as expected according to the requirement?
How does the function handle the scenario when is 1? the output as expected according to
requirement?
8. Can this function handle negative inputs for and j? If not, how can it be modified to handle such cases?
9. What is the maximum recursion depth for this function, and how does it depend on the input values and
10. Are there any edge cases scenarios that this function might not handle well? If so, can you suggest any modifications to improve the code?
This is an example of GPT3.5-turbo v.s. GPT4 for tracing question generation recursion java problem.
D Zero-Shot vs Few-Shot in GPT4
Zero-Shot Prompt... Requirement: (blank) Code:

---

Requirement: (blank)
Code:
public static int redo (int i, int j)
if (i==0)
return 0;
else
return redo (i/i, +)+1;
Few-Shot Prompt... Here are some more examples of tracing questions based on code snippet:
1.
Requirement: (blank)
Code:
for (int i 3; i <= 9; i++)
System. out. print (""*"");
Tracing question:
How many times does the following method print a x?
2.
Requirement: (blank)
Code:
//precondition: x >=0
public static void mystery (int x)
System. out. puint (x $ 10);
if ((x / 10) != 0)
mystery (x 10);
System.out out.print (x & 10);
Tracing question:
Given the following method declaration, which of the following is + printed as the result of the call mystery (1234)
3.

---

Requirement: (blank)
Code:
public static int product (int n)
if (n <= 1)
return 1;
else
return product (n 2);
Tracing question:
Given the following method declaration, what value is returned the result of the call product (
Your goal is not just evaluate their coding skills, but to instill deep, lasting understanding of Java programming in them. By asking relevant tracing questions, you're helping to nurture new generation of critical thinkers, problem solvers, and proficient programmers. You will ask many relevant tracing questions given 1. Requirement, Code, as below. You don't have to strictly follow the examples, and we encourage you to be creative when thinking about the tracing questions: Requirement: (blank) Code:
Requirements (blank)
Code: public static int redo (int i, int j)
if (i==0)
return 0;
else
return redo (i/i, +1;
Zero-Shot Result:
1. What is the purpose of this function? What does it aim to accomplish?
What is the base case for this recursive function? What happens when the base case is reached?
3. If the input values are == 8 and] = 2, what will be the output of the function? Please trace through the code step-by-step and explain your reasoning.
4. If the input values are i = 27 and j = 3, what will be the output of the function? Please trace through the code step-by-step and explain your reasoning.
5. What happens if the input values are i = and] j 5? Does the function handle this case correctly?
Explain your reasoning.
6. What happens if the input values are and = 0? Does the function handle this case correctly according to the requirement? Explain your reasoning.
7. What happens if both input values are negative, such as i = -8 and -2? Does the function handle this case correctly according to the requirement? Explain your reasoning.
8. Are there any edge cases potential issues with the current implementation of the function? If what are they and how can they be addressed?
Few-Shot Result:
Tracing question 1: What is the output of the method redo(9, 3)?
Tracing question 2: What is the output of the method redo(0, 5)?
Tracing question 3: What is the output of the method redo(27, 3)?
Tracing question 4: How many times does the method redo call itself when 16 and j = 2?

---

Tracing question 5: Can this method handle cases where either i or is a negative number? If so, what would be the output for redo(-6, 2)?
This is an example GPT4 in Few-Shot setting v.s. Zero-Shot Setting.


Reviews:
[{"review_id": "2A43cCjeB2", "reviewer": "Reviewer_t9s4", "summary": "", "questions": "A. Please elaborate more details in Section 4.3, that is, does BLEU refer to BLEU4? What does BERTScore mean?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer summarizing both the potential and limitations observed in our study. As a focused contribution, our work made the following key contributions:\n\n* The curation of a high-quality dataset consisting of education experts and LLM-generated code tracing questions and associated code snippets.\n* An exploration and evaluation of GPT4’s capability in question generation, including comparisons with both GPT3.5-turbo and human-authored questions, and an examination of few-shot and zero-shot scenarios.\n* The introduction of a human evaluation methodology and a comprehensive assessment of the quality of LLM-generated questions, offer valuable insights into the potential of LLMs in educational contexts.\n\nThrough multi-faceted evaluations, our results demonstrated that LLMs can produce tracing questions comparable to human-authored ones in quality, while exhibiting greater diversity. The low discernibility highlights LLMs' ability to generate questions that experts could not reliably distinguish from human-authored questions. This provides evidence that LLMs can create tracing questions independently rather than just paraphrasing human questions. We believe these findings meaningfully demonstrate the potential of LLMs for this underexplored application.\n\nWe also acknowledge the study's limitations, including the focus on a specific dataset, the lack of personalized recommendations, and the absence of classroom evaluations. As you rightly noted, these aspects represent interesting future work that can build on the foundations we have established here. As a short paper, our scope was a focused analysis that acts as a first important step towards incrementally addressing these limitations in follow-up studies. \n\nPlease do not hesitate to suggest any other future directions that you think could be worthwhile.\n\nQuestions Answered:\n\nQ1: In Section 4.3, Does BLEU refer to BLEU4? \n\nAnswer: Yes, the BLEU metric used in Section 4.3 refers specifically to BLEU-4. This is the 4-gram BLEU score, which calculates n-gram precision for up to 4-grams. We use nltk sentence_bleu for implementation. We would be happy to add these clarifications to our manuscripts \n\nQ2: What does BERTScore mean?\n\nAnswer: We appreciate these questions and would be happy to add additional information to our manuscript to clarify the meaning and the use of BERTScore. BERTScore measures the similarity between two texts based on their BERT embeddings. It uses a pre-trained BERT model to extract contextual embeddings for each input text, and then calculates similarity using precision, recall, and F1.\n\nSpecifically in the paper, BERTScore was used in two ways:\n1. Section 3.3. To compare the diversity of GPT-3.5 and GPT-4 - In Figure 2, We draw a box plot of BERTScores for the many generated questions with regard to one reference. GPT-4's box plot showed a wider distribution of BERTScores. This indicates GPT-4 produced questions with more lexical diversity compared to GPT-3.5, reflecting GPT-4's ability to generate more diverse questions.\n2. Section 4.3. BERTScore was used to compare the semantic similarity between the GPT-4 generated questions being evaluated in this study and the reference human-authored questions. In Table 4, The moderate BERTScore F1 of 0.303 suggests the GPT-4 questions capture similar semantics but with different text than the human questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The work well demonstrates how to prompt LLMs for a series of code-tracing questions when given a snippet of codes.\n* When the generated data is released, it may bring benefits to the field of programming education and dialogue systems.\n", "reasons_to_reject": "* The experiment results show the potential and limitations of the LLMs-generated questions for code-tracing. \n", "questions_to_authors": "A. Please elaborate more details in Section 4.3, that is, does BLEU refer to BLEU4? What does BERTScore mean?", "paper_topic_and_main_contributions": "The study uses large language models (LLMs) to generate code-tracing questions for learning a programming language. Results demonstrate that GPT4 has an acceptable ability and potential to generate questions for code-tracing and help programming education. The generated data can be one of the contributions to the field of programming education and developing a dialogue tutoring system.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 1 and Table 2 can be merged."}, "extra_text": {}}, {"review_id": "jCWdoU2Je1", "reviewer": "Reviewer_TK2P", "summary": "", "questions": "See reasons to reject", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review and feedback on our manuscript! We would like to address some of the concerns raised and provide further clarifications:\n\n1. Our choice of GPT-4 was based on it being the current state-of-the-art model for code generation and understanding, as evidenced by leaderboards like HumanEval (https://paperswithcode.com/sota/code-generation-on-humaneval). Our initial experiments found GPT-4 generated more diverse questions than GPT-3.5 turbo for the same prompts, as measured by BERTScore (Section 3.3, Figure 2). Examples in Appendix D illustrate GPT-4's greater range of tracing questions. While we only evaluated GPT-4, future work could incorporate comparisons to other recently released models. We chose GPT-4 because of its superior performance on code understanding and also because it does not require local computing resources, making it easily accessible for education practitioners, or researchers who would like to explore this topic.\n\n2. We appreciate your suggestion regarding the personalization of tracing questions based on student submissions. In this initial study, our focus was to compare LLM-generated questions with expert-authored ones from established curricula like CSAwesome. This provides a crucial baseline for quality and methodology that we believe sets the stage for future personalization work. Insights from this foundational study will inform upcoming efforts to tailor questions to individual student code submissions, as well as practical studies involving learner engagement. Thank you for highlighting this important avenue for research.\n\n3. We acknowledge the importance of practical testing scenarios, such as classroom experiments for this type of educational work. Our initial analysis focused on the assessment of the questions’ quality as we believe this is a necessary step before practical testing can be implemented by deploying LLM-generated questions in the classroom. For future work, our ongoing project focuses on the design and construction of an education platform implemented with tracing questions generated by LLMs, usability testing with students, and an evaluation of learning outcomes. We consider a real classroom setting as the immediate step 2 after this study.\n\n4. (and 5.) To comprehensively evaluate the quality of generated questions, we adopt both automatic evaluation with a variety of metrics and human evaluation with objective and subjective ratings. Evaluation metrics used in this study: \n* Automatic/Objective Evaluation: 1) BERTScore similarity metric comparing GPT-4 and GPT-3.5 questions for the same prompts. This showed GPT-4 had higher diversity. 2) Textual similarity metrics like BLEU, ROUGE, and BERTscore compared GPT-4 questions to human questions. \n * Subjective Evaluation: 1) Expert preferences for best questions when comparing GPT-4 and human-authored pairs. 2) Discernibility of authorship using confusion matrix of expert guesses. 3) Quality ratings by experts on relevance, clarity, difficulty, etc. using a standardized rubric. \n\nWe selected our evaluation metrics based on established literature, including BLEU (Post, 2018) for automatic evaluation. Our automatic evaluations assessed textual similarity and diversity through metrics like BLEU, ROUGE, and BERTScore. The human evaluation criteria roughly followed (Kurdi et al., 2020) to cover multiple aspects like relevance, clarity, difficulty appropriateness, and quality. For the purpose of our study, we further split relevance into 1) relevance to code snippets, and 2) relevance to learning objectives. In the perspective of quality, “is this a tracing question” was used to evaluate the validity of a tracing question. These were rated on a standardized 1-5 rubric by 4 experts, who were screened with criteria such as being computer science graduate students with at least 1 year of programming teaching/tutoring experience.\n\nIn summary, our work establishes the preliminary steps in a novel and underexplored task. While we recognize areas of potential extension, we believe that the findings presented in this manuscript are an important first step in demonstrating LLMs' capacity for automated code-tracing question generation. These insights are not only valuable for researchers but are immediately implementable by educational practitioners in their instructional projects.\n\n\n### References:\n\n[1] Post, M. (2018). A Call for Clarity in Reporting BLEU Scores. WMT 2018, 186.\n\n[2] Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, S. (2020). A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30, 121-204.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The code-tracing scenario proposed by this paper is novel.\n- A dataset has been constructed for this scenario.\n- An exploration of LLM's application in this scenario.\n- The metrics has been established to evaluate the quality of the generated code-tracing questions.", "reasons_to_reject": "- The study only considered GPT-4 and has not incorporated other models.\n- There is no personalized question recommendation based on different people's submissions, which shows a lack of thorough exploration of the scenario.\n- The study lacks attempts to use this approach in practical testing scenarios.\n- The metrics for evaluating code-tracing questions are not comprehensive enough.\n- The setting of the metrics is somewhat subjective and lacks an automatic evaluation.", "questions_to_authors": "See reasons to reject", "paper_topic_and_main_contributions": "This paper investigates the application of LLMs for outputing code-tracing questions in indroductory prgramming courses.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5ZzXWVvlUI", "reviewer": "Reviewer_Qaqj", "summary": "", "questions": "Here are some questions I would have for the authors:\n\n- You tested GPT-4 in a zero-shot setting without any demonstration questions. How would an few-shot approach comparing a wider range of prompting strategies affect the quality and diversity of generated questions?\n\n- The human evaluation relied on subjective quality ratings. Were any objective measures like correctness considered? How can you further validate the appropriateness of questions?\n\n- Have you considered testing this approach on other programming languages besides Java or other question types besides tracing? How robust is the approach to new domains?\n\n- Were there any noticeable differences in the types of errors or limitations seen in GPT-4 questions compared to human questions?\n\n\n- Do you have any thoughts on potential risks like teaching incorrect programming concepts if generated questions contain bugs? How can this be addressed?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review and feedback on our manuscript! Below we respond to your helpful questions and suggestions.\n\n### Response to weaknesses:\n- Small data size: We appreciate you raising the important point about the current dataset size. As an initial study, our focus was an analysis of introductory Java tracing questions, especially those in a publicly available curriculum (CSAwesome). While expanding the dataset breadth and size is critical for future work this staged approach allows us to evaluate the methodology before larger classroom integration. Moreover, the evaluation expert screening process, requiring at least 1 year of programming teaching experience, helped mitigate the smaller dataset size. All 4 experts were computer science graduate students who received compensation for their thorough evaluation. Based on the promising results so far, we plan to apply our methodology to larger datasets with more diverse question types. This staged approach will allow us to evaluate the approach and ultimately integrate it into classrooms. \n\n- Limited scope: We intentionally focused on introductory Java tracing questions as a controlled initial domain to study. This allowed exploration of the nuances and challenges specific to code tracing. Now that we have established the methodology and initial findings, expanding to other programming languages and question types is a logical next step. Much of the human annotation protocol, training approach, and evaluation framework should be generalized to new domains with small adaptations. We agree broadening the scope will be important for future work to understand generalizability across different types of educational questions.\n\n### Response to questions:\n(1) Few-Shot Prompting Approach: We appreciate the reviewer's suggestion to further explore few-shot prompting strategies. In preliminary experiments, we found that providing example questions introduced a bias that limited diversity compared to zero-shot prompting (see examples in Appendix D). Our initial human evaluation of the few-shot questions showed substantially lower diversity compared to zero-shot, with performance comparable to template-based methods.\n\n(2) Correctness metrics: In our study, we incorporated a metric where our experts evaluated whether each generated item qualified as a 'tracing question.' This was an important step in confirming the legitimacy of the generated questions in this particular category. While the overall quality of these generated tracing questions was found to be high, we opted not to overburden our expert evaluators with multiple criteria for correctness. Instead, we introduced a 'discernibility' metric, aiming to investigate whether the questions could be distinguished as either human-authored or generated by an LLM. We believe that the results of this discernibility metric could offer insights into the quality and applicability of the generated questions. The evaluation form also contains quality ratings by experts on relevance, clarity, difficulty, etc. using a standardized rubric. \n\nIn terms of established automatic metrics, our methodology employed well-regarded measures such as BLEU (Post, 2018), ROUGE (Lin, 2004), and rubrics from (Kurdi et al., 2020). These measures aimed to assess key attributes like question quality, diversity, and relevance. We also utilized textual similarity metrics for automatic evaluation to gauge the diversity of generated questions. If you have any specific recommendations on such metrics, we would be deeply appreciative of your expert advice.\n\n(3) New Domains: Testing the approach on other programming languages and question types is an excellent suggestion. We focused on intro Java tracing questions as an initial study, but evaluating robustness to new domains will be important in future work. We can reuse much of the methodology and infrastructure from this paper when expanding the scope.\n\n(4) Differences in questions: We acknowledge that human-authored and LLM-generated questions exhibit distinct characteristics. As discussed in our discernibility study (Section 4.2, Table 3), it was very difficult for expert annotators to discern whether individual questions were authored by humans or GPT-4. However, when analyzing question quality holistically (Table 1), human-authored questions were more likely to receive high scores. This suggests nuanced differences exist between the question types. To deeply assess these differences, we have initiated a separate study focused on comparing the types and quality of LLM-generated and human-authored questions. While this analysis is still underway, preliminary results reveal insights into the limitations and biases of LLM question generation. Your question underscores the importance of this direction, and we aim to provide a detailed characterization of the differences in a follow-up work. \n\n(5) Risks: You raise a critical point about potential risks like propagating incorrect concepts if LLM questions contain bugs. Rigorous vetting by experts is an important safeguard we should adopt before any real-world deployment. A system that deploys such questions in a classroom environment should also include checks using test suites and static analysis to catch bugs prior to student exposure. Minimizing the risk of students learning incorrect information is an ethical priority we agree is necessary to explore.\n\nWe appreciate these insightful suggestions for strengthening the work. We will work towards incorporating this valuable feedback to improve our methodology and analysis in future extensions. Please let us know if we can address any other questions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper has several strengths that would make it a valuable contribution to the NLP community:\n\n- Novel application area: This is the first work exploring LLMs for code tracing question generation, an important educational application distinct from prior NLP work. Expanding LLMs to this new domain is an interesting direction.\n\n- Rigorous human evaluation: The paper introduces a thoughtful evaluation methodology including multiple human ratings and textual similarity metrics. This provides a model for human evaluation of LLM-generated content. \n\n- High-quality dataset: The human question dataset compiled is a unique asset that can enable further research. Releasing this data would be very beneficial.\n\n- Insightful analysis: The paper provides a nuanced analysis around the promise but also limitations of LLMs for this application. The insights into quality, diversity, and discernibility of LLM questions are impactful.\n\n- Interdisciplinary relevance: This work sits at the intersection of NLP, education, and human-AI collaboration. Demonstrating the potential of LLMs for enhancing education would be appealing to a broad audience.\n\n- Practical implications: With further development, the proposed techniques could lead to LLM integration in educational platforms and classrooms. This could significantly aid programming education and learning.\n\nPresenting this work would expand the NLP community's understanding of LLMs' capabilities on a non-traditional task. It would also highlight an interesting new application domain with practical relevance. The novel human evaluation framework and high-quality dataset are additional assets that could catalyze further work in this direction. Overall, this is a timely exploration that would be well-suited for an NLP venue.", "reasons_to_reject": "Some potential weaknesses and risks to consider:\n\n- Small dataset size: The dataset has only 176 question pairs from limited sources. A larger, more diverse dataset could strengthen the evaluation and generalizability of findings.\n\n- Limited scope: The focus is narrow - only on intro Java programming tracing questions. Testing breadth by targeting other domains/question types would be informative.", "questions_to_authors": "Here are some questions I would have for the authors:\n\n- You tested GPT-4 in a zero-shot setting without any demonstration questions. How would an few-shot approach comparing a wider range of prompting strategies affect the quality and diversity of generated questions?\n\n- The human evaluation relied on subjective quality ratings. Were any objective measures like correctness considered? How can you further validate the appropriateness of questions?\n\n- Have you considered testing this approach on other programming languages besides Java or other question types besides tracing? How robust is the approach to new domains?\n\n- Were there any noticeable differences in the types of errors or limitations seen in GPT-4 questions compared to human questions?\n\n\n- Do you have any thoughts on potential risks like teaching incorrect programming concepts if generated questions contain bugs? How can this be addressed?", "paper_topic_and_main_contributions": "This paper explores the potential of using large language models (LLMs) to automatically generate code-tracing questions for introductory programming courses. Here are the key points:\n\n- Problem addressed: Manually creating good code-tracing questions is time-consuming and lacks scalability. Existing automated methods using program analysis are limited. This paper investigates whether LLMs can generate high-quality and diverse code-tracing questions.  \n\n- Approach: The authors prompt GPT-4 to generate tracing questions based on code snippets and descriptions. They curate a dataset of human-authored questions for comparison. \n\n- Metrics: A set of human evaluation metrics are introduced to assess quality of LLM questions versus human questions in terms of relevance, clarity, difficulty, etc. Textual similarity metrics are also used.\n\n- Findings: Analysis shows GPT-4 can produce questions comparable in quality to human-authored ones. Over 50% of LLM questions were mistaken for human questions by experts. GPT-4 questions were sufficiently distinct from human questions.\n\n- Contributions:\n\n1) A high-quality dataset of human and LLM tracing questions. \n2) Methodology and analysis to evaluate LLM's ability for this task.\n3) Insights into the potential of LLMs for educational content generation.\n4) A foundation for considering LLMs' inclusion in learning platforms.\n\nIn summary, this paper explores and evaluates the promise of LLMs for automatic high-quality code-tracing question generation, an important problem in computer science education. The human evaluation framework and dataset are valuable contributions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No important references appear to be missing.", "Typos_Grammar_Style_and_Presentation_Improvements": "Overall the paper is very well written. I did not notice any major grammatical or typographical errors in the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers find that this paper offers valuable contributions by exploring the potential of LLMs in code-tracing question generation for programming education, backed by a robust methodology and insightful analysis. It addresses a novel application area with interdisciplinary relevance, making it a strong candidate for acceptance in an NLP venue.

The authors raised some concerns with the review submitted by Reviewer 1, which does not seem to be of good quality (the scores are low, but the actual reasons to reject are missing). Moreover, the reviewers highlight reasons to reject such as lack of personalization and narrow scope (Java programming), which have more to do with the limitations of the paper (potential future work) than with the soundness of the described methodology.

Paper Topic And Main Contributions:

This paper investigates the potential of using large language models (LLMs), specifically GPT-4, to automatically generate code-tracing questions for introductory programming courses. The study addresses the challenge of time-consuming and limited scalability in manually creating such questions and explores whether LLMs can provide a solution. The authors employ GPT-4 to generate questions based on code snippets and descriptions, comparing them to human-authored questions using various evaluation metrics. Notably, the results reveal that GPT-4 can produce questions of comparable quality to those crafted by humans, with over 50% being indistinguishable from human-authored questions. The contributions of this research include a high-quality dataset of both human and LLM-generated code-tracing questions, a rigorous evaluation methodology, insights into the potential of LLMs for educational content generation, and a foundation for considering their integration into learning platforms. In essence, this paper offers a promising solution to the challenge of automating code-tracing question generation in computer science education, providing a valuable dataset and evaluation framework in the process.

Reasons To Accept:

* Novelty and Benefits for Educational Applications
    * The paper introduces a novel application of LLMs in code-tracing question generation, offering a unique perspective distinct from previous NLP work.
    * The generated data, when released, has the potential to bring significant benefits to the field of programming education and dialogue systems, addressing an important educational need.
* Quality of the Methodology
    * The authors have constructed a high-quality dataset tailored to the code-tracing scenario, which can serve as a valuable resource for future research.
    * The introduction of a rigorous evaluation methodology, including human ratings and textual similarity metrics, sets a standard for assessing LLM-generated content, contributing to the field's evaluation practices.
* Quality of the Analyses
    * The paper provides insightful analysis, delving into the promise and limitations of LLMs in generating code-tracing questions, offering valuable insights into quality, diversity, and discernibility.

Reasons To Reject:

* The study solely focuses on GPT-4 for code-tracing question generation and does not incorporate other LLMs, limiting the exploration of potential alternatives and comparative analysis.
* The dataset used in the study consists of only 176 question pairs from limited sources, which raises concerns about the statistical robustness and generalizability of the findings. A larger and more diverse dataset is needed for a more reliable evaluation.
* The metrics used for evaluating code-tracing questions are not comprehensive enough, and the setting of these metrics is somewhat subjective, lacking an automatic evaluation component.
Decision: Accept-Findings