Answer-state Recurrent Relational Network (AsRRN) for Constructed Response Assessment and Feedback Grouping
Authors: Zhaohui Li
Susan E Lloyd
Matthew D Beckman
Rebecca J. Passonneau
Venue: EMNLP 2023 Findings (2023)
Keywords: Writing assessment
relation networks
contrastive learning

Abstract: STEM educators must trade off the ease of assessing selected response (SR) questions, like multiple choice, with constructed response (CR) questions, where students articulate their own reasoning. Our work addresses a CR type new to NLP but common in college STEM, consisting of multiple questions per context. To relate the context, the questions, the reference responses, and students' answers, we developed an Answer-state Recurrent Relational Network (AsRRN). In recurrent time-steps, relation vectors are learned for specific dependencies in a computational graph, where the nodes encode the distinct types of text input. AsRRN incorporates contrastive loss for better representation learning, which improves performance and supports student feedback. AsRRN was developed on a new dataset of 6,532 student responses to three, two-part CR questions. AsRRN outperforms classifiers based on LLMs, a previous relational network for CR questions, and few-shot learning with GPT-3.5. Ablation studies show the distinct contributions of AsRRN's dependency structure, the number of time steps in the recurrence, and the contrastive loss.

Answer-state Recurrent Relational Network (AsRRN) for Constructed Response Assessment and Feedback Grouping
Zhaohui Li"
Susan E. Lloyd Matthew D. Beckman Rebecca J. Passonneau'
DDepartment of Computer Science and Engineering, Pennsylvania State University (zj15282, rjp4 9) edu Department of Statistics, Pennsylvania State University
(se15591, mdl2 68 @psu edu
Abstract
on SR items, where assessment immediate, This conficts, however, with the current emphasis in STEM on critical thinking. reasoning and writing (Graham et al., 2020; Birenbaum and Tatsuoka, 1987). Manual assessment of CR is time consum- ing, and more so when feedback comments ac- company score. We present a relational neural network that can be adapted to handle different CR formats, and that outperforms strong baselines on new dataset of statistics questions developed measure statistical reasoning at the introductory college level. Moreover, also groups responses for similar feedback, in an unsupervised way.
STEM educators must trade off the case of assessing selected response (SR) questions, like multiple choice, with constructed response (CR) questions, where students articulate their reasoning. Our work addresses CR type new to NLP but common in college STEM, con- sisting of multiple questions per context. To re- late the context, the questions, the reference re- sponses, and students' answers, we developed an Answer-state Recurrent Relational Network (AsRRN). In recurrent time-steps, relation vec- tors are learned for specific dependencies in computational graph, where the nodes encode the distinct types of text input. AsRRN incorpo- rates contrastive loss for better representation learning, which improves performance and sup- ports student feedback. AsRRN was developed on a new dataset 6,532 student responses to three, two-part CR questions. AsRRN outper- forms classifiers based on LLMs, a previous relational network for CR questions, another graph neural network baseline, and few-shot learning with GPT-3.5. Ablation studies show the distinet contributions of AsRRN's depen- dency structure, the number of time steps in the recurrence, and the contrastive loss.
The most well known and reasonably sized NLP datasets for CR questions are the BEETLE and SciEntsBBank data from SemEvval 13 Task 7 (Dzikovska et al., 2013) and the Kaggle dataset ASAP (Shermis, 2015). The former has 253 prompts across 16 STEM domains; the latter has 10 prompts where the STEM mostly from biol- ogy, with 22,000 items. Assessment scales range from 2- 2-w to 5-way. There are often multiple reference answers for each point level. ASAP also contains scoring rubrics. In both datasets, the prompts are standalone questions, but there are many other CR formats (Livingston, 2009; Ben- nett, 1991). We present ISTUDIO, new dataset with context-based questions, a common format in STEM. Table shows a context, its two ques- tions, the rubric and reference answers for the first question, and student response to each question. Recent automatic assessment of CR questions us- ing neural models has achieved good performance on the benchmark datasets mentioned above. The standard approach is to encode the question prompt and student answer separately, and send their con- catenation to classifier output layer. Riordan et al. (2017) achieved good performance on ASAP with biL.STM. Later work used transformers (e.g.. BERT (Devlin et al., 2019)). Using model that en- codes rubric elements, Wang et al. (2019) achieved
1 Introduction
STEM educators must trade off the of assess- ing selected response (SR) questions, like multi- ple choice, with constructed response (CR) ques- tions, where students articulate their own reason- ing. Natural Language Processing (NLP) research has turned increasingly towards assessment of con- structed response (CR) questions in recent years, but models remain relatively simple, in part be- cause datasets either relatively small for neu- ral methods, or proprietary. In addition, public datasets are limited to standalone questions. Timely feedback is important in the context of formative assessment throughout a course. As a result, for- mative assessments in STEM currently rely largely

---

Context 2
Some people who good car for music can identify the notes they hear when music played. One method of note identification consists of music teacher choosing one of seven notes (A. B, E, F, the piano. The student asked to name which note was played while standing the room facing away from the piano so that he or she cannot see which note the teacher plays on the piano.
random and playing
Questions
[2a] Should statistical inference he used to determine whether Carla has a "good car for music"? Explain why you should or should not use statistical inference this scenario.
[2b] Next, explain how you would decide whether Carla has a good car for music using this method of note identification.
Rubric
Student advocates use of statistical inference AND Student provides rationale with accommodation for variability (e.g. repeat test method many times; compare to chance model) OR Student deseribes amalysis of probabllityprooporionnnnnm
for [2a]
of correct- incorrect....
Reference answers
[Correct (2 points)] We should use statistical inference because there could be cases of Carla just getting
lucky if we just count how many times she gets note right.
for [2a]
[Partially Correct (1 point)] would use statistical inference see if Carla has good car for music.
IIncorrect (0 points)] No, there no numerical values which could be applied to "Good car for music". [Student_2 2CK2_ [2]] You could use statistical inference by seeing how many notes she answer correctly and the more she gets then the better "ear for music" she has. (Correct: 2 points)
Student
answers
[Student_ 2CKx, q2b) You could give her a test on the notes and the more she answers correctly, then the better she at music. (Partially Correct: 1 point)
Table 1: An ISTUDIO example with context and two question prompts, For [2a], the rubric for correct is shown (other classes omitted to save space), plus one reference example per three correctness classes. A student response is shown for each question.
results similar to (Riordan et al.. 2017). Saha et al. (2018) separately encoded the question, the refer- ence answer and the student answer using InferSent (Conneau et al., 2017), and combined the three vec- tors with manually engineered features for word overlap and part-of-speech information as input to a simple classifier. On the nine SemEval tasks, ac- curacies ranged from 0.51 to 0.79. Good results are also reported on proprietary datasets (Saha et al., 2018; Sung et al., 2019; Liu et al., 2019). Ca- mus and Filighera (2020) compared several pre- trained models on SeiEntsBank, where RoBERTa (Zhuang et al., 2021) performed best. SFRN (Li et al., 2021) outperformed all these models using a more novel relational architecture. It used BERT to separately encode the question, the reference an- swer (SemEval) or rubric (ASAP), and the student answer, and then learned single relational vector over the three encodings. We use more complex relational network that outperforms SFRN.
Besides automating the assignment of correct- score, NLP has the potential to support feed- back comments. However, when and how to pro- vide such feedback is not well understood, and is more of an unsupervised clustering problem than a classification problem, as illustrated in Figure 1. The figure shows a coincidence matrix of feed- back clusters for 100 randomly selected student responses to ISTUDIO question [2b] labeled par- tially correct, and where answers were partially essentially correct on [2a]. Two statistics experts worked independently cluster the responses, and assign a feedback phrase. Expert identified three clusters that covered all but 4 items (ne for no clus- ter). Expert B originally identified 14 that covered all but one item, then combined clusters that were similar and arrived at 4 clusters. Figure 1 shows an overlap of 43 items from A's and B's largest clus- ters, and another overlap of 21 items. All the other cells have small counts. This shows that different people are likely to provide comments in differ- ent ways, but might agree on a few categories that would cover a large proportion of student answers. To make progress towards automated formative assessment that can group similar responses, we develop a novel use of contrastive learning (Chen et al., 2020a; Chopra et al., 2005), to learn represen- tations that are more similar to a reference answer that distinet from other reference answers.
59
30
5
5
1
Total
Anc
2
1
4
A.6
9
A.2
5
21
26
A.t
43
6
1
53
B.t
8.22
8.63
B.04
B.nc
Relational networks (Palm et al., 2018), where graph nodes are input elements and edges are re- lation vectors, can have complex graph configura- tions, and are more efficient than graph attention or graph convolution (Nastase et al., 2015; Liu and
Figure I: Coincidence matrix of feedback clusters identified hy two statistics experts, A (3 clusters) and B (4 clusters) for 100 student responses to ISTUDIO question 2b (shown in Table I; ";"n"" "ne is for no cluster). The boxed calls show overlap in two clusters, amounting to 64% of the total.

---

Zhou, 2020). We propose ASRRN, family of recurrent relational networks where recurrence is over time, to learn distinet relations among the dif- ferent inputs in CR datasets. Depending on the constituency of a dataset, each distinct input text type (e.g., scenario context, question prompt, refer- ence answer, student answer, rubric) is encoded by pretrained language model, and becomes node in graph network where edges are dependency re- depen-
Relational networks (RNs) were initially devel- oped an alternative other graph-based mod- els for reasoning problems in visual, linguistic, or symbolic domains (e.g.. physics (Santoro et al., 20),), where distinct types of input elements have correspondingly distinct interrelations. RNs have surpassed human performance on the CLEVR vi- sual question answering dataset (Johnson et al., 2017), and have become a general framework for few-shot learning (Sung et al., 2018). Li al. (2021) showed that an RN for short answer assess- ment could efficiently learn relations among encod- ings of the question, reference answer, and student answer. Recurrent Relational Networks (RRNs) (Palm et al., 2018) eschew the flat relational struc- ture of an RN for fully connected relational graph with pairwise message passing. We found this orig- inal RRN structure be disadvantageous for our CR tasks (see discussion accompanying Figure 3), For ISTUDIO, AsRRN adopts a dependency struc- ture that reflects how people reason about the text elements shown in Fig. 1.
lations. We find that the deney structure the one that eaptures the semantic dependencies of the domain. A global state node is a dependent of all other nodes. We apply con- trastive learning (CL) where reference examples serve as positive and negative prototypes. AsRRN outperforms multiple strong baselines on correct- ness classification, and to some degree replicate the feedback grouping illustrated in Figure 1.
2 Related Work
Deep neural networks (DNNs) have been applied to CR assessment at least since (Alikaniotis et al.. 2016). As noted in the introduction, most DNNs for CR encode the (reference answer, student answer) pair and use various output functions. Alikani- otis al. (2016) used Long-Shon Short Term Mem- ory (LSTM) networks enhanced by word-specific scores, and achieved high score correlations on ASAP. Riordan et al. (2017) combined Convo- lutional Neural Networks (CNNs) with LSTMs. Model performance with these earlier DNNs was sometimes shown to improve in combination with hand-crafted features (Saha al., 2018) and text mining (Süzen al., 2020). As noted above, pretrained LMs like RoBERTa have performed well (Camus and Filighera, 2020). We rely on pretrained LMs to encode the input layer, and use a novel relational network.
Contrastive Learning (CL) aims improve rep- resentation learning by making similar data in- stances closer and separating dissimilar ones (Le- Khac et al., 2020). CL can be applied in unsuper- vised or supervised learning, and benefits a wide array of visual, language and multi-modal tasks. Chopra et al. (2005) pioncered the introduction of contrastive loss dimensionality reduction. Sub- sequently, Schroff et al. (2015) proposed triplet loss, which employs an anchor to concurrently han- dle positive and negative samples. Later, Chen et al. (2020b) introduced the highly-regarded self- supervised normalized temperature-scaled cross en- tropy loss (NT-Xent)), with strong results on un- supervised learning in ImageNet. Khosla et al. (2020) extended NT-Xent loss for application supervised learning: CERT (Fang et al., 2020) im- proved pretrained language models (e.g., BERT) using contrastive self-supervised learning at the sentence level. Qiu et al. (2021) and Pan et al. (2022) developed new text classification methods using contrastive learning loss, and SimCSE (Gao et al., 2021) delivered ofiee-of-theant sentence em- beddings semantic textual similarity (STS) tasks. We propose variant of NT-Xent with a distinct treatment for different classes: we assume multi- ple correct reference examples to the same question should similar, whereas different ways to be par- tially correct should be dissimilar (see Figure 1).
Explicit use of rubric information is less com- mon. AutoP achieves performance boost on ASAP through automatic generation regular ex- pression patterns from top-tier answers and rubrics (Ramachandran and Foltz, 2015). Wang et al. (2019) integrate rubric information via word-level attention, finding that the rubric information com- pensated for less data, but did not otherwise boost performance. We found that incorporating rubrics into AsRRN did not improve performance on IS- TUDIO. This may be due to the compositional logic of rubric statements, that can have multiple conjunctions and disjunctions (see Table l).

---

3 Dataset
student response items, ten percent of which was held in reserve for future research. Ninety percent of the remainder (N=6,532) was partitioned in the proportion 8:1:1 into training (N=5,226), valida- tion (N=653), and test (N=653) sets. The test set contains all the responses with labels from multi- ple raters so that at test time, inter-rater agreement of model predictions can be computed with the reliable human labels.
The context-based format is novel challenge for NLP, but is widely used in STEM, e.g., in so- called two-tier assessments that simultaneously address students' content knowledge/skills in tier one and explanatory reasoning in tier two, as in computer science (Yang et al., 2015), mathemat- ics (Hilton et al., 2013), physics (Xiao et al., 2018) and other STEM subjects. Automatic methods for low-stakes formative assessment can support timely feedback to identify which conceptual is- sues to address next, thus can lead to greater student success in STEM. CR questions that ad- dress reasoning skills are preferable to selected response items but burdensome for humans to as- sess. The ISTUDIO data come from an assess- ment instrument for statistical reasoning called The Introductory Statistics Transfer of Understanding and Discernment Outcomes (ISTUDIO) that was found to be educationally valid and reliable (Beck- man, 2015), meaning there is strong evidence that it measures well what intends to measure.
Approach
The ISTUDIO dataset contains four distinct types of textual input: the context C' for a given ques- tion prompt, the question prompt Q, one or more reference answers R for each correctness class C É {0, 1,2), and the student answers A. Among many possible ways to structure the set of encod- ings of each text type-such as concatenation, stack- ing, attention layers across sequences of text types, and graph networks-we hypothesized that semantic relations among the five text types could be struc- tured as dependency graph rooted at C, with the learned representation of each dependent node be- ing directly influenced by its parent, and indirectly by its ancestors. As discussed in detail below, the specific chain of dependencies we hypothesized is shown in Figure 2. Our hypothesis is similar in spirit to the motivation for recurrent relational networks, namely to process chain of interdepen- dent steps of relational inference, as in (Palm al., 2018: Battaglia et al., 2018). Here, recurrence refers to inferential time-steps for the entire graph, rather than recurrence of processing units. Tem- poral recurrence allows the network to iteratively progress towards the full set of relations among all the nodes of the graph (Palm et al., 2018). AsRRN also incorporates super-node that is dependent all other nodes, in (Zhang et al., 2018), stabilize the flow of information in the graph net- work, and to implicitly represent the current an- swer state at each pass over the whole graph. We test our hypothesized relational structure among C, Q, (Brubier) ), R and A through extensive abla- tions of alternative dependency structures, and by comparison to strong baselines that either have a simpler relational structure, or relational struc- ture at all. In this section, we present details on the graph structure, the flow of information within the graph amessage-passing the temporal recur- rence, the output classifier layer, and our training
The ISTUDIO dataset consists of responses from 1,935 students to three 2-part questions. The as- sessment rubric, which uses 3-way scale of (es- sentially) correct, partially correct, incorrect, spec- ifies the scoring criteria and provides example stu- dent answers for each correctness class. The data is de-identified, and the original IRB confirmed that it can be made public for research purposes in its de-identified form. Students' schools of ori- gin are identified as large, medium or small. I We evaluated inter-rater agreement among two of the co-authors (A, B), and graduate student in statis- ties (C). There were 63 items scored by A. B, and C; two additional sets of 63 were scored by A, B, and B, C. These pairs of raters achieved quadratic weighted kappa (QWK)= > 0.79, and Fleiss Kappa showed substantial agreement among A, and C (FK 0.70). The remaining unlabeled data was assessed independently by A, B and C.2
To prepare ISTUDIO for NLP research, we per- formed data cleaning to eliminate non-answers and responses that were unusually long (greater than 125 word tokens). This yielded 7,258 individual
'Li. Z. S. Lloyd, M. D. Beckman, and R. J. Pas- sonneau. ISTUDIO: Statistical Automatic Response Assessment Dataset. Penn State Data Commons. 2023. https//doi..ogg itt.iddi..rr110.2222222 00226208//PMM-V777
"Fleiss Kappa is described in (Fleiss, 1971). QWK applies a quadratic weighting for agreement, partial agreement, and disagreement applicd to Cohen's Kappa (Cohen, 1960).
IIncorporating the rubric text into AsRRN was not helpful.

---

objective that incorporates contrastive learning. AsRRN's relational graph for ISTUDIO has the four layers shown in schematic form in Fig- ure The input layer to each node type relies on a pre-trained model to encode the distinct types of text input at time to- Note that the C, Q, and A layers are shown as consisting of a single node, whereas the R layer has four nodes. This indi- cates that the pretrained encoding of single text imput is used to initialize the layers for C, Q and A whereas for the layer we initialize with multiple, different reference answers for each correctness class. Again through experimentation, we found that four reference answers for each correctness class achieved the best performance. For simplic- ity, the figure shows only four R nodes instead of the full set of twelve per Q (four per class). The arrows showing the dependencies between layers (referred to below as message passing) essentially correspond to relation vectors computed from the hidden states at each node. The super-node $ is a global relational vector that incorporates informa- tion from the nodes that correspond to the distinct text inputs. The figure also illustrates the tempo- ral recurrence for updating the flow of information within the graph.
where the node function g is another MLP, and æq is the original input encoding at node i. Reliance on each node's previous hidden state hę allows the network to iteratively work towards a solution, as described in (Palm et al., 201 8). Further, persistent use of the input feature vector at each time step is similar to the operation of residual network (He et al., 2016), relieving & from the need to remember the input, and allowing it to focus primarily on the incoming messages from (i).
Classification of a student's answer for correct- ness utilizes the hidden state at node A at the final time step t. The output distribution of is:
o² e (1*)
(3)
where θ is learnable function to assign the A node's hidden state to class probabilities.
The training objective includes cross entropy loss for classification performance and a new con- trastive learning loss that exploits the availability of reference answers to improve the representation learning. Contrastive learning aims to reduce the distance between learned representations of posi- tive examples set of positive exemplars, and to increase the distance from set of negative exem- plars (Chopra al., 2005). We adapt the widely used NT-Xent contrastive loss function (normalized temperature scaled cross entropy; see above).
Before describing the information flow, or mes- sage passing, we point out here that AsRRN can be adapted for CR datasets that have different struc- tures. For example, the C' layer can be removed to fit the types of standalone question prompts found in benchmark datasets such as SemEval-2013 Task 7 (Dzikovska et al., 2013) and ASAP (Shermis, 2015). This is how perform AsRRN experi- ments these two benchmarks, reported below. AsRRN's Recurrent message passing over time corresponds to the computation of relational information among nodes within the neural net- work graph structure. In AsRRN, message passing is restricted to pairs of nodes and j at time t, and each message m'y (or relational vector) is:
Here, where we have n reference examples for each of the three correctness classes, we treat the n reference answers for the predicted class as positive exemplars, and the 2n reference answers for the other two classes negative exemplars.
Our CL function depends on the true class label. For our use case, we make three strong assump- tions: 1) that there one way for a student answer to be correct regardless of the exact wording, 2) that there are a few ways, each distinet from the others, for a student's 's answer to be partially cor- rect that will cover many of the answers, and 3) that there many diverse ways to be incorrect. For the correct class, the n reference answers are assumed to be near paraphrases, thus for a student answer 2; whose label is correct, LCL incorporates a term S for the average of its similarities to the correct reference examples:
m = (2/s((11111 (111))
(1)
where f, the message function, is multi-layer perceptron. All messages mfy are dispatched in parallel, including the noder-node. To update each node i, the messages from its set of neighbor nodes (i) are first summed: m = Ejen()) m/y- Each node hidden state hy is updated recurrently:
sim (x,, 2.)))
sim sim Z.
Sj = AAerage
T
where Iy are exemplars from the same class as Di. and Fy are exemplars from the other classes.
h = g (hg 2, m!)
(2)

---

Quecdion 10g
Sha
Time stop
1-1
Figure 2: For ISTUDIO, the AsRRN dependencies are C-+ Q, Q-R, R-A, as shown in the figure. During training, AsRRN message passing along the arrows to update each node. includng global supernode. At the final time step. learnable function @ computes the output probability distribution. The maximum cosine similarity of vectors A to the four reference answers I in the predicted class supports potential feedback.
For items z; predicted to be partially correct incorrect, loss function incorporates term Sj for the maximum of its similarities to the n reference examples in the same class:
where Vij is the true label for the ith sample and jth class, and Pij represents the predicted probability for the ith sample and jth class. N is the number of samples, and C is the number of classes. If sample belongs to class j. Vij is (true label) and otherwise, and Pij is the predicted probability that sample belongs to class j.
sim I,, x}i)
sim Di, Izn))
Sj -Max
This ensures that if there is one partially correct reference answer that the student answer resembles, the learned representation will become closer to that reference answer.
5 Experiments
Four experiments are reported here. In the first, As- RRN significantly outperforms multiple baselines on ISTUDIO, the 3-way unseen answers (UA) sub- set of BEETLE and SciEntsBank, and the 3-way subset of ASAP. The second set compares AsRRN to tfaiin-of-thuughhe prompting GPT-3.5, Third, extensive ablations verify the benefits of multiple design choices in AsRRN, Fourth, we compare As- RRN's grouping of student answers with human grouping for partially correct responses. For baselines, we use four high-performing LMs available on Huggingface (BERT-base-uncased) RoBERTa-base; ConvBERT-base: DistiBBERTT- base-uncased); also SFRN (Li et al., 2021) and benchmark graph neural network model graph attention network (GAT) (Veličković et al., 2017). For the LMs, we concatenate the encodings of the question prompt, reference answer, and student an- swer, and add an MLP output layer followed by softmax. We do not encode the context in part be- cause it sometimes exceeds the input token length and in part because it does not seem to improve performance. We use two versions of SFRN, with BERT encodings at the input layer; we either pass the 4-tuple of the context (C), question (Q), ref- erence answer (R) and student answer (A) the relational layer, or the (Q,R,A) triple (as in (Li et al., 2021). For the GAT experiments, we also
All class predictions use a normalizing term Sy:
Sk sam xx))
yielding two contrastive loss terms, depending on the ground truth class label:
exp(S,)
Lel. = N Σ log cxp(S,) cxp((S))
(4)
exp(S))
Cor-- - N
Σ log erp(S)) exp(S
(5)
In sum, correct predictions are encouraged to be similar to the average of the correct reference ex- amples and dissimilar to the reference examples for partially correct and incorrect. Partially cor- rect (and incorrect) predictions are encouraged to be similar one of the partially correct (or incor- rect) predictions, and dissimilar to the reference examples for the other two classes.
weighted sum of the cross entropy loss and the contrastive loss: =
The full loss function
(1 (1-X) L) + XLCI. where À is hyperparameter that governs the proportion of the two types of loss.
Cross entropy loss is:
C
LCE = N
Yij log(p))
(6)

---

Model
ISTUDIO
BEETLE
SciEntsBank
ASAP
SERT-bbase-uncae
80.44 (78.71. 81.78)
75.21 (73.91, 77.17)
70.74 (69. 19, 71.70)
77.43 (76.33, 78.10)
ToBRRTa-baa
81.30 (77.34, 83.15)
75.78 (73.60, 78.73)
69.43 (66.96, 70.96)
76.28 (75.42, 77.35)
ConvBERT-bor base
80.49 (78.87, 82.54)
75.41 (73.91, 77.33)
69.83 (66.52, 71.74)
78.64 (78.01, 79.01)
79.71 (78.71, 81.47)
75.21 (73.14, 78.57)
69.94 (67.85, 72.00)
76.82 (75.42. 77.35)
GAT (4 heads)
78.48 (76.72, 80.25)
74.30 (72.52, 75.62)
69.31 (68.15, 70.22)
77.17 (76.15. 79.16)
GAT (8 heads)
78.91 (75.34, 81.16)
74.81 (71.9,, 76.86)
67.31 (65.44, 74)
78.41 (77.21, 79.47)
SERN (C,Q.R.A)
81.68 (79.26, 83,47)
N.A.
N.A.
78. 94 (78.02, 80.19)
SFRN (Q.R.A)
82.15 (80,96, 83.83)
77.02 (75.47, 78.88)
71.74 (68.30, 73.59)
78.75 (78.02, 79.40)
AsRRN (-CL)
83,46 (82.15, 84.23)
76.71 (75.47, 77.64)
72.11 (71.41, 73.04)
80.02 (79.14, 81,20)
AsRRN (+CL)
85,26 (83.46, 86.98)
N.A.
N.A.
N.A.
Table 2: AsRRN and hascline accuracies (CIs) three test sets, and QWK on the fourth. SFRN (C,Q.R,A) cells marked N.A.
indicate the dataset no context. AsRRN (+CL) cells marked N.A. are for datasets with only correct reference answers.
Accuracy
Model GPT-3.5: Reb.
2.a
2.b
3,a
3.b
4,0
4.b
Total
47.17
$7.00
29.35
40.59
57.01
74.54
51,12
GPT-3.5: +RefC, +RubC
49.10
32.71
67.88
58.41
50.00
75.45
55.59
GPT-3.5: +2RefAll, +RubAll
56.25
38.31
65.13
64.35
70.17
81.81
62.78
AsRRN
83.92
84.11
92.66
83.16
88.59
89.09
85,26
Table 3: ASRRN model best performance compared with GPT3.5 prompting.
use BERT pretrained embedding as the input of each node, the graph structure we use is the default connections of all C. Q R, and A nodes with 4 or 8 self-attention heads. 4 All results show 95% boot- strap confidence intervals (Efron and Tibshirani, 1986) from 8 runs on the test sets (parallelized over 8 GPU cores). Table 2 shows evidence of stastical differences, with AsRRN outperforming the four LMs. AsRRN (+CL) also outperforms SERN (C.Q.R.A), and nearly so for SFRN (Q. R, A). The AsRRN model not only outperforms the GAT model across all datasets, but it also boasts a shorter running time. This underscores AsRRN's superior efficiency and enhanced reasoning capac- ity. Details on the AsRRN settings are below; train- ing parameters are in appendix A.
fAIl, +RubAll). GPT-3.5 improves in each next condition, and for question 4.b, comes close to the LM performance. Interestingly, the relative perfor- mance by question for GPT-3.5 is quite different from AsRRN, and both are quite different from the relative difficulty for students.
Table above shows that ablation of CL de- grades performance. We conducted multiple ab- lations on ISTUDIO, and investigated the impact of replacing CL with regularization term. We tested the contribution of each AsRRN layer, find- ing that all layers contribute significantly to the overall performance (results not shown due to space constraints). We also tested different numbers of reference answers and found that four worked best. The four reference answers for each of the 6 ques- tions were chosen by the four co-authors (2 NLP researchers, 2 researchers in statistics education) to ensure they were distinct from each other, and covered many training examples. Figure 3 shows the mean accuracy with different values for the temporal recurrence, and different graph structures. The blue line (circles) for AsRRN shows that learn- ing continues to improve over the first two time steps, then degrades. The red line (triangles) for fully connected RRN is worse at all but the first time step. The yellow line (boxes), labeled ran- dom, represents that in each of the parallelized CI runs, for all pairs of nodes i.j (cf. equation 1), random decision was made at the beginning of training whether to include message passing; a full exploration of all combinations of message passing
GPT models often perform well using chain-of- thought reasoning for zero-shot and shot set- (Brosn et al., 2020; Wei et al., 2022), so we tested this method on ISTUDIO, using GPT-3.5. In all conditions in Table 3, showing total accu- racy and accuracies per question, the prompting included the context, the question, the class labels, and a student answer for the entire test set (Ap- pendix D includes details). Results the first row (-Ref, -Rub) are from these inputs alone (similar to zero-shot). In the second row, one correct ref- erence answer and the rubric for the correct class were added (+I,, +RubC). In the third row condition, two reference examples for each class, and the rubric for each class were provided (+2Re-
*Less fully connected configurations performed worse.

---

partially correct class, we selected four reference answers that were distinct from one another. Here we report on how consistent two experts (A and B) were with each other, and each with AsRRN, in pairing a student answer with the same reference answer for questions 2b and 4b; these are two of the three most challenging questions, based on the average student scores. We used QWK report agreement of the three pairs. For both questions, A and worked independently pick one of the four reference answers as equivalent to each stu- dents' partially correct answer, or "none of the above." For the subset of accurate partially correct predictions from AsRRN (90 out of 107 for 2b; 100 out of 110 for 4b), we show QWK for the expert pair, and each expert with ASRRN. Cases where AsRRN's maximum probability class had p < 0.85 were treated as "none of the above" based on loca- tions of sharp decreases of output probabilities in scatterplots.
85-
84
Accurec
83-
82-
Rendam
AsliNNi
0
Time
Figure 3: Mcan test accuracy and Cls with different numbers of reasoning time steps for AsRRN, a fully connected RRN, and an RRN with random message passing.
would have been unnecessarily time-consuming. This condition has lower performance than AsRRN across all but the last time step. Finally, we ex- amined different values of À in the loss function, and compared L1 L.2 regularization. For all values of À performance better than for A (see Appendix C). For À 1, accuracy is still reasonable, because Lc1 implicitly includes the ground truth class. Replacing Lel with an L1 or L.2 regularizer severely degrades performance. Results in Tables 2-3 use 4 reference answers per class, 2 time steps, and A=0.01.
As shown by the QWK scores in Table 4, the two experts had good agreement of 0.72 on question 2b, which is consistent with the pattern shown in the coincidence matrix in Figure 1. They had very poor agreement on 4b, however, which shows that this task is difficult even for experts. On 2b, AsRRN agreement with of 0.69 was close to that for A and B on the partially correct reponses that AsRRN accurately predicted (0.71), but agreement with A was much lower (0.53). On 4b, agreement between A and B AsRRN's accurate subset was 0.45, and here AsRRN had similar agreement with B of 0.41, then very high agreement with A of 0.70. Instruc- tors report that unpredictable with any given set of students what kinds of partially correct an- swers will occur frequently, as depends on many unknowns, such as the students' prior background. Using contrastive learning in a novel way, we have taken first step towards partially automating this difficult but pedagogically important task.
Evidence suggests that timely and specific feed- back helps students learn (Pearl et al., 2012; Garfield et al., 2008). For formative assessment feedback, instructor effort is best spent on partially correct (PC) answers, where there is an opportunity to scaffold students towards more complete under- standing. Incorrect answers are far more diverse, and students may be more confused about the rea- soning steps and less likely to integrate feedback into their thinking. To assess the results on whether AsRRN groups student responses with the same reference answers human would, we find mixed results when we compare two humans and AsRRN the grouping of partially correct answers. Recall that there were four reference answers for all three correctness classes, for all six questions. For the
The high accuracy of AsRRN, and its potentially useful association of student answers with a spe- cific reference answer, argues for the quality of the representation learning. Appendix B includes t-SNE visualizations of the learned representations of the test set with and without contrastive learning.
Test Set
Correct Pred.
Q
A.
Size
A. B
A. RN
RN
Size
26
0.72
107
0.71
0.53
0.69
Conclusion
4b
0.39
110
0.45
0.70
0.41
100
AsRRN is a novel recurrent relational network where the nodes in the neural graph each corre- spond to a distinct type of textual input, a super-
Table 4: For questions 2b and 4b, QWK for experts on the test set; also QWKs on the subset of AsRRN's correct predictions for the experts and AsRRN (RN in the table).

---

node represents global information across all nodes, and the answer state the final time step becomes the input to the final classification layer. The graph structure (direction of message passing). leads to efficient learning of relational information that sup- ports multi-step problems. AsRRN can have more or fewer layers, depending on the number of dis- tinct types of text input in the dataset. A novel loss function for contrastive learning improves clas- sification accuracy, and provides a step towards feedback for instructors and students on partially correct responses, by learning which of multiple reference answers a response is most similar to. A thorough set of ablations provides evidence for the chosen dependency structure, number of time steps in the temporal recurrence, and the contrastive learning. On ISTUDIO, AsRRN significantly out- performs strong baselines.
References
Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek
Rei. 2016. Automatic text scoring using neural net- works. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 715-725, Berlin, Ger- many. Association for Computational Linguisties.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Al- varo Sanzhez- Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. 2018. Relational in- ductive biases, deep learning, and graph networks. arXiv preprint 10xiv:1806.01220
Matthew D. Beckman, 2015. Assessment of cognitive transfer outcomes for students of introductory statis- fics. Ph.D. thesis, University of Minnesota-Twwin Cities).
Randy Elliot Bennett. 1991. On the meanings of con- structed response. ETS Research Report Series, 199 (2):i44.
Menucha Birenbaum and Kikumi K Tatsuoka. 1987. Open-ended versus multiple-choice respouse for- mats-it does make difference for diagnostic purposes. Applied Psychological Measurement. 11(4):385-355
7 Limitations
The AsRRN model relics heavily on manual graph design, based on the relational structure of the CR dataset, and on the quality of reference an- swers. While this limits generalizability, it also incorporates expert domain knowledge useful ways. Additionally, while AsRRN provides a pre- liminary foundation for feedback, understanding of optimal feedback across varying assessment con- texts remains limited. Although the importance of timely and relevant feedback for students is well- established, little is known about the specific forms of feedback that work best in different contexts, or how feedback varies across instructors. While AsRRN can point to a specific reference answer for partially correct student answers, the task definition and AsRRN performance need much improvement.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Leon Camus and Anna Filighera. 2020. Investigating transformers for automatic short answer grading. International Conference on Artificial Intelligence in Education, pages 43-48. Springer.
Ting Chen, Simon Kornblith, Mohammad Norouzi. and Geoffrey Hinton. 2020a. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
The ISTUDIO dataset has relatively few ques- tion prompts (N=6) and is limited to single sub- ject domain, in contrast benchmark datasets that have dozens of prompts and many domains. On the other hand, ISTUDIO derived from a well-validated, reliable instrument and was labeled through an extensive inter-rater agreement study.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597-1607. PMLR.
Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning similarity metric discriminatively, with application face verification. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'0S) volume I, pages 539-546 vol.
8 Acknowledgements
This work was supported under NSF DRK award 2010351 and NSF IUSE award 2236150.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational und psychological mea- surement, 20 1):37466.

---

Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Nat- ural Language Processing, pages 670-680, Copen- hagen, Denmark. Association for Computational Lin-
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Annette Hilton, Geoff Hiltona, Shelley Dole, and Merri- lyn Goos. 2013. Development and application of a two-tier diagnostic instrument to assess middle-years students' proportional reasoning. Mathematics Edu- cation Research Journal, 15:523-545.
guistics.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transfommrrs for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages Minneapolis, Minnesota. Association for Computational Linguistics.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitmick, and Ross Girshick. 2017. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 2901-2911
Myroslava Dzikovska, Rodney Nielsen, Chris Brew, Claudia Leacock, Danilo Giampiccolo, Luisa Ben- tivogli, Peter Clark, Ido Dagan, and Hoa Trang Dang. 2013. SemEval-2013 task 7: The joint student re- sponse analysis and 8th recognizing textual entail- ment challenge. In Second Joint Conference on Lexi- cal and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Work- shop on Semantic Evaluation (SemEval 2013), pages 263-274, Atlanta, Georgia, USA. Association for Computational Linguistics.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola. Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Su- pervised contrastive learning. Advances in neural information processing systems, 33:18661-18673.
Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. 2020. Contrastive representation learning: A frame- work and review, Ieee Access, 8:9999771193934
B. Efron and R. Tibshirani. 1986. Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy. Statistical Science, 1(1):54- -555 75.
Zhaohui Li. Yajur Tomar, and Rebecca J. Passonneau. 2021. A semantic feature- wise transformation re- lation network for automatic short answer grading. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6030-60400.
Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005. 12766.
Tiaoqiao Liu, Wenbiao Ding, Zhiwei Wang. Jiliang Tang, Gale Yan Huang, and Zitao Liu. 2019. Au- tomatic short answer grading via multiway attention networks. In International Conference on Artificial Intelligence in Education, pages 169-173 Springer.
Joseph I. Fleiss. 1971.
Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
Zhiyuan Liu and Jie Zhou. 2020. Introduction to graph neural networks. Synthesis Lectures on Artificial Intelligence and Machine Learning, 14(2):1-127.
Tianyu Gao. Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence beddings. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Samuel A. Livingston. 2009, constructed-responte test questions: Why we use them; how we score them. Technical Report 11, Educational Testing Service.
J. B. Garfield, D. Ben-Zvi, B. Chance, E. Medina, C. Roseth, and A. Zieffler. 2008. Assessment in statistics education. In J. B. Garfield, D. Ben-Zvi, B. Chance, E. Medina, C. Roseth, and A. Zieffler, editors, Developing Students' Statistical Reasoning: Connecting Research and Teaching Practice, pages 65-89. Springer Netherlands.
Vivi Nastase, Rada Mihalcea, and Dragomir R Radev. 2015. A survey of graphs in natural language process- ing. Natural Language Engineering. 21(5))665-698.
Rasmus Palm, Ulrich Paquet, and Ole Winther. 2018. Recurrent relational networks. Advances Neural Information Processing Systems, 31.
Steve Graham, Sharlene A. Kiuhara, and Meade MacKay. 2020. The effects of writing on learning in science, social studies, and mathematics: A meta- analysis. Review of Educational Research, 90(2):79 226.
Lin Pan. Chung Wei Hang, Avirup Sil, and Saloni Pot- dar. 2022. Improved text classification via contrastive adversarial training. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36 (10). pages 11130-11 1338.

---

D. K. Pearl, J. B. Garfield, R. delMas, R. E. Groth, J. Kaplan, H. McGowan, and H. S. Lee. 2012. Connect- ing research to practice in culture of assessment for introductory college-level statisties.
Petar Veličković. Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:
Yao Qiu, Jinchao Zhang, and Jie Zhou. 2021. Improv- ing gradient-based adversarial training for text clas- sification by contrastive learning and auto-encoder.
Tianqi Wang, Naoya Inoue. Hiroki Ouchi, Tomoya Mizumoto, and Kentaro Inui. 2019. Inject rubrics into short answer grading system. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low- Resource NLP (DeepLo 2019), pages 175-182.
In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1198-1707, Online. Association for Computational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arxiv:2201.11903.
Lakshmi Ramachandran and Peter Foltz. 2015. Gener- ating reference texts for short answer scoring using graph-based summarization. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Build- ing Educational Applications, pages 207-212.
Yang Xiao, Jing Han, Kathleen Koenig, Jianwen Xiong, and Lei Bao. 2018. Multilevel Rasch modeling of two-tier multiple choice test: A case study using Law- son's classroom test of scientific reasoning. Physical Review Physics Education Research, 14:020104.
Brian Riordan, Andrea Horbach, Aoife Cahill, Torsten Zesch, and Chungmin Lee. 2017. Investigating neu- ral architectures for short answer scoring. In Pro- ceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 159-168.
Tzu-Chi Yang, Sherry Y. Chen, and Gwo-Jen Hwang. 2015. The influences of a two-tier test strategy on student learning: A lag sequential analysis approach. Computers & Education, 82:366-377.
Swarnadeep Saha, Tejas I Dhamecha, Smit Marvaniya, Renuka Sindhgatta, and Bikram Sengupta. 2018. Sentence level or token level features for automatic short answer grading?: Use both. In International Conference on Artificial Intelligence in Education, pages 503-517. Springer.
Yue Zhang, Qi Liu, and Linfeng Song. 2018. Sentence- state LSTM for text representation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 317-327, Melbourne, Australia. Association for Computational Linguistics.
Adam Santoro, David Raposo, David G Barrett, Ma- teusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. 2017. A simple neural net- work module for relational reasoning. Advances in neural information processing systems, 30.
Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics. pages 1218-1227. Chinese Information Processing Society of China.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for
face recognition and elustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823.
A Training Parameters
Mark D Shermis. 2015.
Contrasting state-of-the-art in the machine scoring of short-form constructed responses. Educational Assessment, 20(1):46-66 65.
We employed the AdamW optimizer for model training, with parameters set to learning_ seatiing_ateeeeeeeeee 5 and weight criihttdecay=0.000 and clipped the global norm of the gradient at 1.0. The state vector of the graph nodes and the message passing vector have hidden dimensions of 768 and 128, respectively. The weight of the CL loss (A) is set to 0.01, and the temperature (r) of CL loss is 1. To curb overfitting, a 20% dropout rate (hidden_dropout prob=0.2) is applied to the hidden layers. The maximum token length for each text data input is set to 64 in BEETLE dataset, 128 in SciEntsBank and ASAP datasets, and 256 in ISTUDIO.
Chul Sung. Tejas Indulal Dhamecha, and Nirmal Mukhi. 2019. Improving short answer grading using transformer-6 based pre-training. In International Con- ference Artificial Intelligence in Education, pages 469-481. Springer.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208.
We use warm-up prior to training with no CL loss during which the learning rate incrementally in- creases from a negligible value to the pre- learn- ing rate. The ger per__inerr_scheiiee schedule withh p
Neslihan Süzen, Alexander N Gorban, Jeremy Leves- ley, and Evgeny M Mirkes. 2020. Automatic short answer grading and feedback using text mining meth- ods. Procedia Computer Science, 169:726-743.

---

function from the Hugging Face Transformers li- brary generates schedule with learning rate that linearly decreases following a warmup period of linear increase. In our experiment, the number of warmup steps is configured to be 5% of the total training steps.
70
All models were trained a server equipped with eight NVIDIA RTX A5000 GPU cores, with the AsRRN model's ss training time ranging from two five hours, depending on the training size and number of epochs. the paper is accepted, we will include github link to the code repository.
0.00081
2.001
0,001
0.01
0.05
0.1
0
02
(A)
Figure Mean test accuracy AsRRN on ISTUDIO with different À values in the loss function. represents the contrastive learning in loss function. Using LI L.2 reg- ularization instead of CL in the combined function degrades performance.
B Contrastive Learning Visualization
C Contrastive Loss versus Regularization
For values of À < 1, Figure shows that the overall AsRRN accuracy is fairly stable. For À 11 accu- racy drops to around 78%, which is still reasonable. This can be attributed to the diminished supervi- sion of the classification loss, and the the fact that the contrastive learning loss employs ground truth labels during training, using the mean or max sim- ilarity to reference examples in the ground truth class.
10
0-
-20
-30
(a) Learned data representation without CL.
D GPT 3.5 Prompting
20
Input
"role" rollersyslen"" "content": "You are helpfu tescher. This is construct-rssprnd assessment task. Clessify of Answer besed on the Context Correct, Partially Correct, Incorrect without
"role" "contenf"
-10
'Context: Some people who have a good ear for music which note the teacher plays on the plano. "Question: [2a] Should statistical inference be used
determine whether Carla has good eer for music?
"Answer: (student
assessed]"
-10
(b) Learned data representation with CL.
Output
"rote": "assistant"
Figure 4: Learned representations without (a) and with (b) contrastive loss, visualized with t-SNE projections into 2D. The color key is green for correct, blue for partially correct. and red for incorrect.
'Carreet Pertially Correct/ incorrect I
Figure Zero-shot learning prompt design for the assess- ment of ISTUDIO data, Input is the prompt message, Output the model responses
T-SNE projections of the representations of the examples in the test the two conditions of omitting the contrastive loss term (4a) versus including it (4a). With the con- trastive learning, the items cluster more tightly into fewer groups.
shown in Figure 4 for
powerful model for generating human-like text. In our experiment, we leveraged GPT 3.5 model with a series of "prompts" based on the ISTUDIO materials (containing Context (C), Question (Q), Reference answer (R), and Answer
GPT 3.5

---

(A)). We articulated these prompts using chain- of-thought strategy. Our GPT-3.5 model was imple- mented using the ChatCompletion API provided by OpenAI, specifically using the gpt-3.5-turbo model with 0.7 model temperature.
As exemplified by [2a] in Figure 6, we designed zero-shot learning prompt template for 5PT-3.5 (-Ref, -Rub) displayed in Table 3. The model was supplied with series of "messages" generate classification results. Each message consisted of role (either 'system', 'user', or "assistant') and the message content associated with that role. In one or few-shot learning scenarios (Figure 7) where reference answers were added by the 'user', we provided historical examples to the input. For in- stance, the one-shot (+I,efC, +RubC) included correct reference answer prompt shown in 7a. In the case of few-shot prompts (+2RefAll, +RubAll), we used six refer- ence answers (two for each class) as depicted in 7b. However, due to space limitations, only three ex- amples are shown. Finally, we ensured the model's output always included correctness label followed by an explanation (all marked in red). This chain- of of-thought prompting method not only enabled im- proved results but also laid the foundation for future research in feedback generation.
example in the
Craie "ycienn" "sendead" "You thaatier The candtret dlss
Chinsh
hear" Commact paccil gaod
Sqaccis lia suriiall te be caler can Gola lic maacc"
(rael nsairan: Comtan: DDerect rtuti coooss olastincal inferno in AMD Staten proides
Nawmhs cu cencudes uleull teeulte oae cole ae Oute hass auno out musc?"
haer teae tnd Your and Dovved cinn
(a) One-shot learming prompt.
Foor "our Tro
Quueeber deoul pare betee tn caler Snathe Cana tes ear tur mancc"
Phor: seseart "cols tardaay connect
cortes pecasss
workd Casch seak?"
"Awww No aere
must mndiivatenn iout derec tnng ory atred itmene seiry
Few-shot learning prompt.
Figure 7: One-shor shot and few-shot learning prompt examples for the assessment of ISTUDIO question [2a].


Reviews:
[{"review_id": "wwOKp6YMn9", "reviewer": "Reviewer_DeHo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for all your comments. We respond to your comments and questions below.\n\n“The novelty of the proposed method is limited.”\n\nLike many published works, AsRRN builds up and combines insights from prior approaches.  It's main novelty is greater interpretability of the models' decisions for an important application.  Three specific innovations we pointed to are:\n\n1. We presented a problem that is novel to NLP, and important for post-secondary education.\n\n2. The final structure of AsRRN, verified through ablation experiments, demonstrates how AsRRN incorporates the domain semantics as prior knowledge to boost performance, an insight that could benefit other kinds of NLP problems.\n\n3. While in general, learned representations are uninterpretable, we introduce greater interpretability of the models' decisions on partially correct items, which is a particular challenge for STEM educators in formative assessment of written work.\n\n\n“The dataset size of this paper seems like small to ensure generalizability.”\n\n\nDomain-specific tasks can have unique constraints that make such large datasets impractical or impossible to collect, but where smaller datasets still have value. Collecting and annotating classroom data, for example, is intrinsically labor-intensive and constrained by various factors such as ethical considerations, quality of annotation, and the classroom environment. Our Col-STAT dataset, with 6,532 entries, is substantial, when compared to other notable datasets that have sizes ranging from several hundred to at most 20K: Mohler (2009): 630, Mohler (2011): 2,273, ASAP-SAS(2012): 18,000, Beetle (2013): 5,652, SciEntsBank (2013): 10,598, Arabic (2014): 610, McDonald (2017): 924, Turkish (2020): 984.\n\nMoreover, as highlighted by the results in Table 2, our model generalizes across different runs, even when different development sets are used. Finally, we believe the dataset's generalizability for STEM Assessment—should not be solely gauged by its size. As discussed in Section 3, the data were meticulously collected from actual classrooms and annotated by domain experts, which adds significant value in terms of accuracy and validity, meaning the prior evidence we cited shows that this assessment measures what it is intended to measure, i.e., statistical reasoning skills. \n\nMohler (2009): Mohler, M., R. Mihalcea. Text-to-text semantic similarity for automatic short answer grading. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 567–575. 2009.\nMohler (2011): Mohler, M., R. Bunescu, R. Mihalcea. Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 752–762. 2011.\nASAP-SAS(2012): Shermis, M. D. Contrasting state-of-the-art in the machine scoring of short-form constructed responses. Educational Assessment, 20(1):46–65, 2015.\nSciEntsBank (2013), Beetle (2013):Dzikovska, M. O., R. Nielsen, C. Brew. Towards effective tutorial feedback for explanation questions: A dataset and baselines. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 200–210. 2012.\nArabic (2014): Gomaa, W. H., A. A. Fahmy. Automatic scoring for answers to arabic test questions. Computer Speech & Language, 28(4):833–857, 201\nMcDonald (2017): McDonald, J., R. Bird, A. Zouaq, et al. Short answers to deep questions: supporting teachers in large-class settings. Journal of Computer Assisted Learning, 33(4):306–319, 2017.\nTurkish (2020): Çınar, A., E. Ince, M. Gezer, et al. Machine learning algorithm for grading open-ended physics questions in Turkish. Education and Information Technologies, 25(5):3821–3844, 2020.\n\n“More analysis on the training/ inference time complexity are needed.”\n\nWe trained AsRRN on an NVIDIA RTX A5000 GPU. On Col-STAT, the training fluctuated between 2 to 4 hours, depending on factors such as the ablation condition, and number of time steps in the relation network recurrence. The inference time was consistently under 3 minutes, emphasizing the model's efficiency. Training times on SciEntsBank ranged from 3 to 5 hours depending on the settings, showcasing that even with a dataset approximately twice as large, there is only a 50% to 25% increase in training time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Overall the paper gives good and clear explanation and implementation of AsRNN. Well-structured and smoothly.\n2. The design choices are well-justified with explanations as well as as ablation study.\n3. A new solution to the new task for NLP and release a new dataset.", "reasons_to_reject": "1. The novelty of the proposed method is limited.\n2. The dataset size of this paper seems like small to ensure generalizability.\n3. More analysis on the training/ inference time complexity are needed.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work addresses a CR type new to NLP but common in college STEM, consisting of multiple questions per context. Answer-state Recurrent Relational Network (AsRNN) is proposed, which exploits the graph structure to learn temporal dependencies and contrastive learning for better representation learning. A new dataset is proposed.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QmzTI1QV2x", "reviewer": "Reviewer_ui5s", "summary": "", "questions": "What is the state in ‘answer-state’?\n\nIt seems that the partially correct answers have been chosen to be the candidates for providing formative feedback. However, in a learning setting, incorrect answers demand equal scaffolding demand if not greater. Why is feedback for incorrect answers ignored?\n\nOn the formative feedback part, it is observed that AsRNN has good agreement with B on 2b and with A on 4b. But why is this observation an indicator of high accuracy of AsRNN? To illustrate more, A and B have poor agreement in 4b. Good agreement of RN with A does not ensure robustness of AsRNN. Consider a hypothetical situation where A’s annotation is incorrect and B’s annotation is correct. This will result in poor agreement between A and B.  Having good agreement between A and RN indicates that RN follows an erroneous annotator.\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for all your comments. We respond to your comments and questions below.\n\n“There is a lack of discussion regarding architectural decisions. For example, why is the Recurrent Relational Network chosen over other graph neural networks? The notion of ‘Answer-state’ in Answer-State RNN (AsRNN) has never been motivated or illustrated.”\n\nThe choice of model architecture in this project is driven by a variety of factors, including the research problem, the nature of the data, and the specific requirements of the task.The Recurrent Relational Network (RRN) was chosen because it fits the the problem in four ways:\n1. RRNs were developed for tasks that require a model to reason about relationships among multiple entities or concepts, where the learned relationships are represented as vectors with their own parameters. In contrast to attention networks, the number of parameters is kept low, so models are efficient. In our context, these relationships are among the variables C, Q, R, and A.\n2. Recurrence over time helps RRNs continue to learn complex relationships. In complex scenarios that require temporal relationships to be modeled—as in our task where sequences like C->Q->R->A are evaluated and then loop back to reexamine C—the recurrent structure proves invaluable. \n3. Palm et al. (2018) use a RRN to answer the questions about the relationships between objects. This reasoning capacity makes RRN well-suited to our problem, which demands reasoning about multiple relationships among the various types of text input.\n4. Implementing an RRN from scratch is quite straightforward, because the conceptual structure and the implementation (from Palm et al.) are easy to understand, and it is also straightforward to develop many variants of an RRN from the same codebase. For example, the many ablation conditions required relatively few changes.\nComparison with Other Graph Neural Networks (GNNs):\nWe initially considered alternative GNN architectures such as Graph Convolutional Networks (Kipf et al. 2016, Wei et al. 2016) and Graph attention networks (Veličković et al. 2016). These methods employ neural network layers to aggregate information from a node's neighbors to update node representations directly. For example to update the representation (h_i) of node i, GCN uses h_i = W * (h_neighbor_1+ … … +  h_eighbor_n) + B * h_i, where W and B are learnable parameters. GNNs are well-suited for tasks in computer vision that need to learn one node’s representation, based on a high volume of neighboring nodes, and where the neighbors often represent different regions of the same depicted object (e.g, trunk of a tree). In contrast, our task pertains to a small number of total nodes that each represent a distinct kind of textual element (e.g., question prompt and a reference answer). Instead of merging information across similar neighbors through convolution, our goal is to learn the distinct way that a reference answer relates to a question prompt, and that a student answer relates to a reference answer. We believe RRNs are a good fit to our problem because they compute “messages” (i.e., the relation vectors) in one step, and then use the message to update the node representations (see equations (1) and (2) in our paper).  \n\nKipf, Thomas N., and Max Welling. \"Semi-supervised classification with graph convolutional networks.\" arXiv preprint arXiv:1609.02907 (2016). \n\nWei, Shih-En, et al. \"Convolutional pose machines.\" Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2016.\n\nVeličković, Petar, et al. \"Graph attention networks.\" arXiv preprint arXiv:1710.10903 (2017).\nPalm, Rasmus, Ulrich Paquet, and Ole Winther. \"Recurrent relational networks.\" Advances in neural information processing systems 31 (2018).\n\nClarifying 'Answer-state':\nWe chose the  term “Answer state” because our design choices aim for learning the best possible representation of the ‘answer’ node, through appropriate dependencies with the other nodes, and in particular with the supernode, which essentially represents an implicit global “answer state” at each iteration of the graph. The final output, as well as the application of contrastive learning techniques, is centered on the answer node's state.\n\n“The contributions of the would have not been articulated clearly”\n\nThe final paragraph of our introduction states our contributions, but does not provide a succinct listing, e.g., as bullets. We can reframe this paragraph to state three key contributions: 1) The creation of a novel dataset with input jointly from statistics educators and NLP researchers, specifically designed for the automatic assessment of constructed-response (CR) questions that might have complex dependencies (e.g., on scenario contexts); 2) The development of an Answer-State Recurrent Relational Network (AsRRN) capable of evaluating CR questions of various types that is both effective and highly efficient at learning domain dependencies, such as that the meaning of a question prompt can depend in part on a scenario context; 3) The introduction of a new training paradigm that leverages contrastive learning to enhance performance and to provide greater interpretability of its decisions when student answers are partially correct, given that there are different ways for a student’s answer to miss the mark.\n\n“The architectural novelty is incremental. Only adaptation over RNN seems to be using problem specific dependency rather than considering full connectivity.”\n\nOur work is non-incremental in that we present two innovations that extend the inherent semantic flexibility and computational efficiency of relation networks, and a third innovation that leads to greater interpretability of model decisions. \n\nWe apply RNNs to a semantically complex problem with five distinct categories of candidate text input (scenario context, question prompt, rubric reference answer, student answer), and experimentally demonstrate that an RNN that uses only four of these (all but the rubric; see response to reviewer 3), and where the RNN dependencies capture the domain relations among these text elements, outperforms a fully connected network both in accuracy, and in efficiency.  While we did not save this information we could rerun the comparison and report the lower accuracy & efficiency of the fully connected model.\n\nWe introduce a super-node that is interconnected with all other nodes. This serves to stabilize the information flow throughout the graph network while also implicitly learning a global representation of the current answer state during each iteration over the entire graph.\n\nBecause our training objective not only employs cross-entropy loss to optimize classification performance, but also incorporates a contrastive learning loss, our model can provide insight into decisions about student answers classified as partially correct, through representations that are relatively closer to one of several distinct ways to be partially correct. This latter element takes advantage of the availability of reference answers to enhance the quality of representation learning. Additionally, use of the Max() operator to push the learned representations towards one of a set of partially correct reference answers is an innovation that has not been used in previous versions of contrastive loss objectives.\n\n\n“It seems that the partially correct answers have been chosen to be the candidates for providing formative feedback. However, in a learning setting, incorrect answers demand equal scaffolding demand if not greater. Why is feedback for incorrect answers ignored?”\n\n\nOur collaborators in statistics education note that in the large introductory college-level statistics courses they are familiar with, questions used for formative assessment are intended to identify how well students have learned the concepts presented so far, and that written feedback is much more effective for students who provide partially correct answers, and at the same time much more time consuming for instructors.  Students whose answers are correct have learned the contact, and any written feedback would be limited to something like “Good job.” Students whose answers are incorrect without an indication of partial understanding similarly tend to receive uniform feedback, which is that they need to revisit the relevant material and to get extra help.  Students whose answers are partially correct are best served when the feedback points out the part they got correct, the part they got incorrect, and how to think about ways to avoid the same misunderstanding in the future.\n\n“On the formative feedback part, it is observed that AsRNN has good agreement with B on 2b and with A on 4b. But why is this observation an indicator of high accuracy of AsRNN? To illustrate more, A and B have poor agreement in 4b. Good agreement of RN with A does not ensure robustness of AsRNN. Consider a hypothetical situation where A’s annotation is incorrect and B’s annotation is correct. This will result in poor agreement between A and B. Having good agreement between A and RN indicates that RN follows an erroneous annotator.\n”\n\n The point of this analysis is to illustrate that accuracy is not the relevant criterion, because expert assessors do not always agree, as our data shows.  However, when expert assessors do agree, it is an opportunity to demonstrate that AsRRN can be considered as another expert whose judgment sometimes agrees well with humans who agree, and sometimes does not. Future work will explore ways to build upon AsRRN's ability to utilize partially correct reference answers.\n\n“The symbol (\\tao) in formulation of S_j and S_j^\\prime is not defined.\n”\n\nThe symbol τ (\\tao) represents the hyperparameter for temperature in Contrastive Learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work addresses an interesting and different aspect of automated assessment with respect to existing ASAG tasks and datasets. This task includes constructed response while the other ASAG task definitions mostly cover definition or concept completion type of questions. \n\nThe work also addresses the problem of selecting formative feedback. This aspect of grading has by and large been ignored in ASAG task definition of the earlier works.\n\nThe experiments include comparison with state-of-art LLMs on existing datasets in ASAG domain.\n", "reasons_to_reject": "There is a lack of discussion regarding architectural decisions. For example, why is the Recurrent Relational Network chosen over other graph neural networks? The notion of ‘Answer-state’ in Answer-State RNN (AsRNN) has never been motivated or illustrated. \n\nThe contributions of the world have not been articulated clearly\n\nThe architectural novelty is incremental. Only adaptation over RNN seems to be using problem specific dependency rather than considering full connectivity. \n", "questions_to_authors": "What is the state in ‘answer-state’?\n\nIt seems that the partially correct answers have been chosen to be the candidates for providing formative feedback. However, in a learning setting, incorrect answers demand equal scaffolding demand if not greater. Why is feedback for incorrect answers ignored?\n\nOn the formative feedback part, it is observed that AsRNN has good agreement with B on 2b and with A on 4b. But why is this observation an indicator of high accuracy of AsRNN? To illustrate more, A and B have poor agreement in 4b. Good agreement of RN with A does not ensure robustness of AsRNN. Consider a hypothetical situation where A’s annotation is incorrect and B’s annotation is correct. This will result in poor agreement between A and B.  Having good agreement between A and RN indicates that RN follows an erroneous annotator.\n", "paper_topic_and_main_contributions": "The paper presents an automated method for evaluating constructed responses from the students. The specific assessment situation addressed in this work includes a prompt (context) against which two questions are asked. The first question seems to be an affirmative type while the second question asks for an explanation behind the answer in the first part. The data contains rubric and reference answers for three categories: Correct, Partially correct and incorrect. The authors advocated for feedback grouping. A recurrent relational network is used on a computation graph formulated considering the dependency between elements (context, question, reference answer etc.) present in the evaluation bundle. Several baselines have been used to compare the performance of the proposed model.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The symbol (\\tao) in formulation of S_j and S_j^\\prime is not defined."}, "extra_text": {}}, {"review_id": "22bPvwYcQE", "reviewer": "Reviewer_tuki", "summary": "", "questions": "- Question A: why did not the author show the results using rublic? To avoid the suspicion of arbitrarily changing the experimental setup, they should have first shown the results of the experiment using all input types and then the ablation results.", "limitations": "", "rating": 3, "confidence": 1, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for all your comments. We respond to your comments and questions below.\n\n“The usefulness of the proposed recurrent neural network is unclear because the main factor of performance improvement is the introduction of contrastive learning (CL). Table 2 shows that the proposed method without CL slightly outperformed the baseline SFRN in three out of the four evaluation datasets, but some of them are not significant based on the 95% confidence intervals shown in Table 2. “\n\n\n“It would have been helpful to understand the usefulness of the combination of the graph-based recurrent neural network and contrastive learning in the proposed method if the results of contrastive learning in a simpler setting (e.g., contrastive learning is used when all input types are simply concatenated) had been included, but such results were not included in the paper.”\n\nThe results for AsRRN without contrastive learning (CL) and the two SFRN models are not statistically significantly different, as their confidence intervals overlap. The case remains that AsRRN (+CL) significantly outperforms all the LLM models, while none of the other relation networks significantly outperforms them. It is a great point that we should have applied CL to all the baselines. With SFRN, this would mean prior to the fusion operation, and we could communicate with the authors about revising the codebase that way. With the LLM baselines, it would be difficult to incorporate CL because we concatenated the text inputs.\n\n\n“The difficulty in mapping the symbols (e.g., C_0, C_1, etc.) in Figure 2 to those in the equations makes it difficult to understand the details of the proposed method. For example, the authors did not explain what C_i, Q_i, R_i and A_i in Figure 2 mean in the main context, making it difficult to know how to compute each node in Figure 2. In addition, to my understanding, S_i in Figure 2 is different from the calligraphic S_j (line 431), but if so, how to compute S_i was not explained in this paper.”\n\nThank you for highlighting these issues. It appears we did not adequately explain the notation in Figure 2. In this figure, C_i, Q_i, R_i and A_i correspond to the components described in the Graph Building section of the paper the context (C), question (Q), reference answer (R), and the answer to be graded (A_i), respectively.\nThe x_i in the equations serves as the node representation for A_i, while the other nodes C, Q, R are represented by h_j and x_j etc.\n\n“The details of how the hyper-parameters of the proposed method were determined are unclear. For example, when predicting the final output using the proposed method, it is necessary to fix the time step t in advance, but how to determine t was not explained. If this t was determined using the validation data, such details should be noted in the paper to present that the experiments were properly conducted.”\n\nDetails about the hyper-parameters are in appendix A, and the time-step analysis is in Figure 3. Because time-step=2 had the best performance, we use t=2 at inference time. \n\n“It is difficult to understand STEM-related technical terms such as ISUDIO. Since most EMNLP readers are NLP experts but most of them are not experts of education research, technical terms specific to education research should be appropriately explained with examples. Particularly, the ITUDIO contexts should be explained with examples because no one can imagine the examples. In addition, I cannot understand what the difference is between \"student response\" (line 251) and \"student response items\" (line 289). To my understanding, the authors collected 1,935 \"student responses\", but after data cleaning, they obtained 7,258 \"student response items\". It is not clear why the data cleaning increased the data size.”\n\nISTUDIO is the name of the assessment, i.e. the set of questions. A student whose data is included in the ISTUDIO assessment results responded to all six question prompts, and there were 1,935 distinct students who submitted responses.  If all responses were complete and usable, this would give 11,610 individual item responses, but due to students who did not complete all questions, and data cleaning, the final count was 7,258 items.\n\n“Question A: why did not the author show the results using rublic? To avoid the suspicion of arbitrarily changing the experimental setup, they should have first shown the results of the experiment using all input types and then the ablation results.”\n\n\nWe can add the ablation condition where we included use of the rubrics, where the result was (81.21) with a confidence interval of (78.14, 82.74). As we stated in the paper, this does not improve performance; in fact, it degrades performance.\n\n“The full name of the proposed method (i.e., \"Answer-state Recurrent Relational Network\") only appeared in the Abstract. It also should be written in the main text of the paper.”\n“The font size in Figure 2 is too small to understand what the figure means. In addition, the positions of Figure 2 and its description are too far apart.\n”\nGood point, so we would improve the figures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The research motivation and the relationship between background and the proposed method are appropriately written in this paper.\n\n- Although the performance is not much improved when contrastive learning is not used, the proposed graph-based approach may provide good inspiration for other researchers, ", "reasons_to_reject": "- The usefulness of the proposed recurrent neural network is unclear because the main factor of performance improvement is the introduction of contrastive learning (CL). Table 2 shows that the proposed method without CL slightly outperformed the baseline SFRN in three out of the four evaluation datasets, but some of them are not significant based on the 95% confidence intervals shown in Table 2. It would have been helpful to understand the usefulness of the combination of the graph-based recurrent neural network and contrastive learning in the proposed method if the results of contrastive learning in a simpler setting (e.g., contrastive learning is used when all input types are simply concatenated) had been included, but such results were not included in the paper.\n\n- The difficulty in mapping the symbols (e.g., C_0, C_1, etc.) in Figure 2 to those in the equations makes it difficult to understand the details of the proposed method. For example, the authors did not explain what C_i, Q_i, R_i and A_i in Figure 2 mean in the main context, making it difficult to know how to compute each node in Figure 2. In addition, to my understanding, S_i in Figure 2 is different from the calligraphic S_j (line 431), but if so, how to compute S_i was not explained in this paper.\n\n- The details of how the hyper-parameters of the proposed method were determined are unclear. For example, when predicting the final output using the proposed method, it is necessary to fix the time step t in advance, but how to determine t was not explained. If this t was determined using the validation data, such details should be noted in the paper to present that the experiments were properly conducted.\n\n- It is difficult to understand STEM-related technical terms such as ISUDIO. Since most EMNLP readers are NLP experts but most of them are not experts of education research, technical terms specific to education research should be appropriately explained with examples. Particularly, the ITUDIO contexts should be explained with examples because no one can imagine the examples. In addition, I cannot understand what the difference is between \"student response\" (line 251) and \"student response items\" (line 289). To my understanding, the authors collected 1,935 \"student responses\", but after data cleaning, they obtained 7,258 \"student response items\". It is not clear why the data cleaning increased the data size.", "questions_to_authors": "- Question A: why did not the author show the results using rublic? To avoid the suspicion of arbitrarily changing the experimental setup, they should have first shown the results of the experiment using all input types and then the ablation results.", "paper_topic_and_main_contributions": "This work proposed an answer-state recurrent neural network to improve the quality of assessment of constructed response, which is a kind of scoring task for a given several inputs. In the proposed method, each input type (e.g., context, question, student answer, etc.) is encoded by a pre-trained language model, and the encoded results are fed into a variant of the recurrent network to make the states more suitable for the final prediction. In the training of the network, contrastive learning loss is introduced in addition to general cross entropy loss. To evaluate the proposed network in more STEM-specific setting, the authors created a new dataset Col-STAT, which consists of five input types, i.e., scenario context, question prompt, reference answer, student answer, and rubric. The evaluations were conducted using Col-STAT, BEETLE, SciEntsBank and ASAP datasets and the results showed that the proposed network with contrastive learning achieved the best results on Col-STAT and the proposed method without contrastive learning achieved slightly better performance than the baseline SFRN in three out of four datasets. The authors also compared their results with those obtained GPT-3.5.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The full name of the proposed method (i.e., \"Answer-state Recurrent Relational Network\") only appeared in the Abstract. It also should be written in the main text of the paper.  \n\n- The font size in Figure 2 is too small to understand what the figure means. In addition, the positions of Figure 2 and its description are too far apart."}}]

Author rebuttal:

Meta-review: The reviewers find the task interesting and the paper well-written. However, the reviewers raised some issues with the size of the dataset used; the lack of discussion about architectural decisions; and some found the STEM-specific terminology difficult to understand. 

Overall, the reviewers find the soundness to be borderline, trending good, but are generally ambivalent about seeing the work published at EMNLP.
Decision: Accept-Findings