Automatic Feedback Generation for Short Answer Questions
Using Answer Diagnostic Graphs
1,2,♣
1,2
1,2
1,2Momoka Furuhashi,Hiroaki Funayama ,Yuya Iwase , Yuichiroh Matsubayashi ,Yoriko Isobe 2, Toru Nagahama 1, Saku Sugawara 3, Kentaro Inui 4,1,2♣ furuhashi.Momoka.p4@dc.tohoku.ac.jp

1 Tohoku University, 2 RIKEN , 3 National Institute of Informatics, 4 MBZUAI

Goal : Generate feedback to promote the understanding of logical structure of prompt text

Abstract

Backgroundl Develop a system that uses Answer Diagnostic Graphs (logicalstructure graphs) to generate feedback on students responses
to reading comprehension questions.
l Conduct a practical experiment with high school students toevaluate the potential effectiveness of our feedbackl Individual feedback is necessary because of the variety of errors instudents’ responses.
l Since only scores are returned, it is difficult for students tounderstand the reasons for errors in their answers.
l Answer Guides for review are not individualized.

Short-reading Comprehension Questionsl Prompts require students to explain reasons or details fora specific sentence in the prompt text

Prompt ： 傍線部(3)「…世界と私の間に橋がl Students derive appropriate responses by understanding
the logical structure of the prompt text

Prompt text iscritical essay.

Idea
l By graphing the sentence structure, it is able toclarify the difference between student’s
responses and the correct answer. [Roehling+, 2017]l The errors in students’ responses can be classified
into several types.

Student response：言語によって何かをかかってしまうので，距離も生まれてしまう．」とあるが，どういうことか．本文に即して句読点と共に70字以内で説明せよ．

表現することは...世界との生々しい一体感を失うこと．

(Expressing something throughlanguage means that ... it leads
to losing the vivid sense of unity
with the world.)(What does mean in the phrase”...a bridge is built between theworld and me, distance iscreated as well."? Please answer
in 70 characters.)

I don't understandwhat is wrong
with my response.

BeforeWith our feedback
8/15

I understandwhere to focus inprompt text.

Returns
by score
only.

Our System: Automatic Feedback Generation using Answer Diagnostic Graphs
Students’ responses

Answer Diagnostics Graphsa part of theprompt text

Expressing something...
is losing a raw sense ofoneness with the world.

Response node

【Templates】

Model answer
Losing directconnection
with the world.

Response node
The feeling ofalienation fromthe world.

Rubric criteria available:“““You are able to read element of{the model answer} appropriately.”””Relationshiplabel
Response node
Response node

Figurative expression:“““The referenced paragraph{paragraph number},“ {justificationcues} ”corresponds to a figurativeexpression....”””

Losing a raw
sense of onenesswith the world.

The node estimatedby our system

Feedback

The referenced paragraph 9,“losing a raw sense of onenesswith the world ” corresponds toa figurative expression....

Empirical Study
Experimental Settingl Participants : 41 Japanese high school students

Answer explanation cond. :only answer guide
Feedback cond. :answer explanation & feedback

Divided equally into twogroups based on total scores.

l Two reading comprehension questions

Al Oracle feedback

B

What we research

A. Feedback cond.

A. Answer
explanation cond.

B. Answer
explanation cond.

B. Feedback cond.

l Whether our feedback can help participants understandlogical structure of the prompt text.

QuestionnaireFuture Workl Whether participants can be motivated to revise their responses.

Results and Challengesl How do you set up a bias-free environment?
l What type of experimental setting is best?

The use of our feedback helped participants understand why their responses
were wrong and where to focus on in prompt text.
There was no significant difference in score improvement between answer
explanation cond. and feedback cond. when re-answering.
No significant differences in understanding of sentence structure occurredin the answer explanation and feedback conditions
Feedback was unusual for participants, so there may have been a positive bias.
Edulearn2024

l For example, comparative experiments, etc.

l What kind of feedback is effective for reading
comprehension questions?
l What kind of feedback is effective to promote
understanding of logical structures?
l What metrics are good for evaluating
engagement?