SlowGuess/NAACL_2022_OCR huggingface dataset:
Multi-Relational Graph Transformer for Automatic Short Answer Grading†
Rajat Agarwal Varun Khurana* Karish Grover* Mukesh Mohania Vikram Goyal Indraprastha Institute of Information Technology, Delhi
{rajata, varun19124, karish19471, mukesh, vikram}@iiitd.ac.in
The recent transition to the online educational domain has increased the need for Automatic Short Answer Grading (ASAG). ASAG automatically evaluates a student's response against a (given) correct response and thus has been a prevalent semantic matching task. Most existing methods utilize sequential context to compare two sentences and ignore the structural context of the sentence; therefore, these methods may not result in the desired performance. In this paper, we overcome this problem by proposing a Multi-Relational Graph Transformer, MitiGaTe, to prepare token representations considering the structural context. Abstract Meaning Representation (AMR) graph is created by parsing the text response and then segregated into multiple subgraphs, each corresponding to a particular relationship in AMR. A Graph Transformer is used to prepare relation-specific token embeddings within each subgraph, then aggregated to obtain a subgraph representation. Finally, we compare the correct answer and the student response subgraph representations to yield a final score. Experimental results on Mohler's dataset show that our system outperforms the existing state-of-the-art methods. We have released our implementation<sup>1</sup>, as we believe that our model can be useful for many future applications.
Grading student work is critical for assessing their course understanding. However, answer grading can become monotonous and tedious for teachers. Automatic Short Answer Grading (ASAG) task is to assign ordinal scores to student answers, given
Figure 1: A motivating example for using multiple relations in automatic short answer grading.
some 'model' answer by an academician or instructor. Figure 1 presents a sample question, model answer, and student answers from an undergraduate computer science course (Mohler et al., 2011). One of the early approaches for solving the ASAG task has been to build models based on human-designed features (Mohler et al., 2011; Sultan et al., 2016). Recent works utilize deep learning methods such as convolutional neural networks (CNNs), long short-term memory networks (LSTMs), Transformer (Vaswani et al., 2017) to learn the representation of student responses and to avoid designing features manually (Alikaniotis et al., 2016; Hassan et al., 2018; Kumar et al., 2017; Riordan et al., 2017; Yang et al., 2018). Due to semantic heterogeneity, the main problem in assessing student responses given instructor-provided model answers is a complex natural language understanding task (the same answer could be articulated in different ways)(Gomaa et al., 2013).
We hypothesise that a student's answer will be considered correct if the keywords in the answer are in the right relationship with the corresponding
† This research work is supported by Extramarks Education India Pvt. Ltd. (an education technology company). Last Author acknowledges the support ofInfosys Center of AI IIITD and iHuB Anubhuti.
<https://github.com/kvarun07/asag-gt>
Human Language Technologies, pages 2001 - 2012
words from the model answer. As can be seen from Figure 1, Student A is awarded full marks because the words like Testing Phase and Coding are adequately associated with words in the model answer like testing stage and coding phase. However, other students are awarded partial marks because not a lot of words in those correspond to some relation in the model answer, i.e. the decreasing score corresponds to the decreasing number of relations in the model answer being captured in the student response. This motivates us to incorporate structural relationship context information for effective comparison. We discuss the various types of relations captured in detail in the further sections.
This paper applies the principle of short text matching to solve the problem of grading short subjective student's response. The key steps for textual matching are efficient textual representation, followed by semantic matching. In literature, we see that short text matching is broadly based on two approaches: sequence-based and structure-based. Sequence-based models fully exploit semantic information of sentences without incorporating syntactic information (Mueller and Thyagarajan, 2016), (He et al., 2015; Cer et al., 2018; Conneau et al., 2017; Agirre et al., 2014). Recent works by (Vashishth et al., 2019; Croce et al., 2011; Severyn et al., 2013) have found that the structural information of sentences is beneficial for sentence representation. Therefore, structure-based neural networks (Le et al., 2018; Yao et al., 2019; Huang et al., 2019), (Defferrard et al., 2016; Liu et al., 2020) exhibit better performance than sequence-based models. Graph Convolutional Network (GCN) (Kipf and Welling, 2016) can extract semantic and syntactic information of sentences simultaneously from the graph. GCN first propagates information among nodes and their neighbors and then provides node representation by aggregating the received information. However, GCNs are designed for homogeneous graphs and cannot handle different types of nodes and relationships in the graph. Recently there have been attempts to explore relationships in the graphs. Schlichtkrull et al. (2018) introduces RGCN to handle relationships in knowledge graphs by using specific matrices for each relationship. Nevertheless, it focuses only on characteristics of the relations and does not study different types of features associated with a node.
This paper introduces a Multi-Relational Graph Transformer (MitiGaTe) for ASAG to incorporate
the structural context. We first transform a sentence into an Abstract Meaning Representation (AMR) graph (Banarescu et al., 2013). AMR parses a sentence into a rooted directed graph. Then subgraphs are prepared corresponding to the relationships (types of edges) in the original AMR graph. For each subgraph, MitiGaTe prepares relation-specific token representations and aggregates them to obtain a subgraph representation. Finally, these relation-enriched subgraph representations of the student and the model answer are matched using multi-perspective matching (Wang et al., 2017) and the matching result yields the student score. We evaluate our model on the benchmark Mohler's dataset (Mohler et al., 2011) and it outperforms the current state of the art models.
1. We propose a Graph Transformer-based technique to incorporate relation-enriched structural information for ASAG.
2. We also demonstrate that including the semantic representation of a relationship in the preparation of token embeddings improves the model's overall performance.
3. We perform a case study to show that MitiGaTe can provide reasonable feedback to students explaining the (in)correct parts of the student answer.
4. MitiGaTe is evaluated through extensive experiments on a benchmarking dataset. The experimental results verify our proposed model's performance.
ASAG: Traditional methods utilize handcrafted features, such as lexical similarity features (Dzikovska et al., 2013), graph alignment features (Mohler et al., 2011), n-gram features (Heilman and Madnani, 2013), soft cardinality text overlap features (Jimenez et al., 2013), averaged word vector text similarity features (Sultan et al., 2016) and other shallow lexical features (Ott et al., 2013). More recently, deep learning approaches have been utilized for the automatic short answer scoring task. Mueller and Thyagarajan (2016) proposed a siamese adaptation of the LSTM network for labelled data comprised of pairs of variable-length
sequences. Zhao et al. (2017) proposed an efficient memory networks-powered automated scoring model. Riordan et al. (2017) explored simple LSTM and CNN-based architectures for short answer scoring. Kumar et al. (2017) proposed a method involving Siamese biLSTMs, a novel pooling layer based on the Sinkhorn distance between LSTM state sequences, and a support vector ordinal output layer. However, the approaches mentioned above do not incorporate the structural information, and as a result, the matching performed is partly inadequate (Croce et al., 2011; Severyn et al., 2013).
Application of GCN on NLP: GCN is a simplified graph neural network (GNN) introduced by (Kipf and Welling, 2016) to perform semi-supervised classification. In NLP, GCN is mainly explored in tasks such as semantic role labeling (Marcheggiani and Titov, 2017), machine translation (Bastings et al., 2017). Yao et al. (2019) first model a whole corpus as a graph where documents and words are regarded as nodes. However, most GNNs were designed for homogeneous graphs and could not handle different nodes and relations in heterogeneous graphs.
Unlike the existing methods, we consider the role of relations to improve the learning of more fine-grained node representation.
We formally define the ASAG short text matching problem as follows: Given two sentences  $\mathcal{A}^M = \{w_1^M, w_2^M, \ldots\}$  and  $\mathcal{A}^S = \{w_1^S, w_2^S, \ldots\}$ , where  $\mathcal{A}^M$  and  $\mathcal{A}^S$  refers to Model and Student answer respectively with  $w$  as words in the sentence, the goal of a text-matching model  $\mathbf{f}(\mathcal{A}^M, \mathcal{A}^S)$  is to compute the semantic similarity of  $\mathcal{A}^M$  and  $\mathcal{A}^S$ .
In this section, we discuss a graph-based matching model. To create graphs from the input sentences, we first parse each sentence into an AMR graph (Section 3.1). Further, we prepare subgraphs  $G_{sub}$  from AMR graphs corresponding to each relation (Section 3.2). The intuition behind subgraph splitting is to get relation-enriched structural information which can improve matching performance and interestingly can be used to provide a reasonable feedback to students (Section 5.5). We then create relation-specific token representation  $\mathbf{h}_{w,r}$  from each subgraph and aggregate them to a final subgraph representation denoted as  $g_{r,M}^{\phi}$  for model and  $g_{r,S}^{\phi}$  for student answer (Section 3.3). Lastly,
we compare them in Section 3.4 to predict the grading score in Section 3.5.
As shown in Figure 3, our model consists of five layers namely, Text to AMR conversion, Subgraph preparation layer, Graph Transformer Encoder layer, Subgraph matching layer and lastly, score prediction layer. We discuss each layer below.
3.1 AMR Parsing
The meaning of a sentence is represented by AMR as a rooted directed graph. Here, nodes represent the concepts or predicates and are not always directly related to words. Edges in AMR represent the relations between concepts such as subject/object. AMR provides a high-level abstraction by capturing meaningful content but ignores functional and inflectional words in a sentence (Xu et al., 2021).
We choose AMR over dependency parser for sentence parsing because unlike the dependency structure of a sentence where each word token is a node in the dependency tree, AMR concepts and relations abstract away from actual word tokens. Content words generally become concepts while function words either become relations or get omitted if they do not contribute to the meaning of a sentence, which is more intuitive and suitable for the ASAG task, unlike dependency parser that merely extracts grammatical relations between entities. Further, the AMR parser parses semantically similar but syntactically dissimilar answers into nearly similar graphs, which ensures that students who answer differently are not penalised.
We use the AMR Model  $\mathrm{API}^2$  from amrlib library to create AMR graphs  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  of a given input sentence  $S$ . Each node  $v\in V$  in the AMR graph represents a concept or predicate. Edge  $e_{i,j}$  denotes the specific relation type between nodes  $v_{i}$  and  $v_{j}$ . The details are discussed in Section 4.2. AMR Graphs for student answers in Figure 1 are shown in Appendix A.
3.2 Subgraph Preparation Layer
We transform the original AMR graph into subgraphs based on the number of relations or types of edges in the graph. All the subgraphs have the same number of nodes as the original graph. However, only a particular type of edge is enabled, and
$^{2}$ https://bit.ly/amrlibrary
Figure 2: Example to show subgraph splitting and intuition behind using AMR for ASAG task. Similar relations in model and student answers have similar structures. Example, location and store in model answer are connected as  $A_0$  and similarly block and hold in student answer are connected with  $A_0$ . This applies to other relations as well.
all other types of edges are disabled in the subgraph corresponding to relation  $r$ .
We first group all edge types into one to get a homogeneous subgraph referred to as the default subgraph default. The default subgraph is an undirected graph that contains the complete connected information in the original graph. Then we split the input graph into multiple subgraphs according to the edge types. Figure 2 demonstrates the subgraph preparation by an example.
AMR uses approximately 100 different relations to capture the semantics. Thus, it would be inefficient to capture all these relations in separate subgraphs as many of these occur rarely. In this work, we have used ARG1 and ARG0 relations to capture primary information, and all remaining relations are grouped as an other relation. We will be denoting ARG1 and ARG0 relations as  $A_{1}$  and  $A_{0}$  for the scope of this paper.
With reference to the PropBank<sup>3</sup> guidelines, the  $A_0$  label is assigned to arguments which are understood as agents, causers or experiencers. The  $A_1$  label is usually assigned to the patient argument, i.e. the argument which undergoes the change of state or is being affected by the action. The other category could include relations for quantities like :unit, date-entities like :time, and semantic relations like :consist-of. More information about the various types of relations captured by AMR can be found in the original paper (Banarescu et al., 2013).
Hence, we can denote the collection of these subgraphs as  $G_{sub}$ , where,
$$
G _ {s u b} = \{d e f a u l t, A _ {0}, A _ {1}, o t h e r \} \tag {1}
$$
3.3 Preparing Node and Subgraph Representation
In this layer, we prepare a relation-specific subgraph representation that reflects the characteristics of tokens in a particular relation. We perform two steps: firstly, we prepare relation-specific node representation using Graph Transformer and secondly, all relation-specific node representations are aggregated into a relation-specific subgraph representation. Below we discuss the process for tokens in  $\mathcal{A}^M$  and the same process is applied to  $\mathcal{A}^S$ .
3.3.1 Relation-Specific Node Representation
Our model is adapted from the Transformer model introduced by (Vaswani et al., 2017). It is a sequence-to-sequence neural architecture originally used for neural machine translation. It uses encoder-decoder architecture. The encoder consists of two sublayers: a self-attention mechanism and a position-wise feed-forward network. The self-attention mechanism employs  $H$  attention heads, and each of them learns a distinct attention function. Finally, the outputs of all attention heads are concatenated, followed by feed-forward layers, residual connections and normalization. The encoder computes the token representations iteratively using the output of the previous layer as input.
3https://verbs.colorado.edu/~mpalmer/projects/ace/PBguidelines.pdf
Figure 3: The framework of the proposed model MitiGaTe employs Graph Transformer for learning the structural context of a sentence. Please refer to Section 3 for more details.
Transformer (Vaswani et al., 2017) treats the sentence as a fully-connected graph. In MitiGaTe, we mask the non-neighbor nodes' attention while updating each node's representation. Specifically, we mask the attention  $w_{ij}$  for node  $j \notin \mathcal{N}_i^+$ , where  $\mathcal{N}_i^+$  is the set of neighbors of node  $i$  in the graph including self-loop.
So given the input sequence  $x = (x_{1},\ldots ,x_{n})$  the output representation of node  $i$  denoted as  $h_i^{l + 1}$  for  $l + 1$  th layer is computed as follows:
$$
h _ {i} ^ {l + 1} = O _ {h} ^ {l} \left\| _ {k = 1} ^ {H} \left(\sum_ {j \in \mathcal {N} _ {i} ^ {+}} w _ {i j} ^ {k, l} V ^ {k, l} h _ {j} ^ {l}\right) \right. \tag {2}
$$
$$
e _ {i j} ^ {l + 1} = O _ {e} ^ {l} | | _ {k = 1} ^ {H} \left(\hat {w} _ {i j} ^ {k, l}\right) \tag {3}
$$
$$
w _ {i j} ^ {k, l} = \operatorname {s o f t m a x} _ {j} \left(\hat {w} _ {i j} ^ {k, l}\right) \tag {4}
$$
$$
\hat {w} _ {i j} ^ {k, l} = \left(\frac {Q ^ {k , l} h _ {i} ^ {l} \cdot K ^ {k , l} h _ {j} ^ {l}}{\sqrt {d _ {k}}}\right) \cdot E ^ {k, l} e _ {i j} ^ {l} \tag {5}
$$
where  $Q^{k,l}$ ,  $K^{k,l}$ ,  $V^{k,l}$ ,  $E^{k,l} \in \mathbb{R}^{d_k \times d}$ ,  $O_h^l$ ,  $O_e^l \in \mathbb{R}^{d \times d}$  are trainable parameter matrices,  $k = 1$  to  $H$  denotes the number of attention heads and  $||$  denotes concatenation. Following (Dwivedi and Bresson, 2020) we explicitly incorporate edge representation  $e_{ij}^l$  to improve attention weights  $w_{ij}^{k,l}$ . This above mentioned process of Graph Transformer is applied for each relation  $r \in G_{sub}$ . For brevity, we denote node representation of the last layer as  $h_{w,r}$
where  $w$  represents a word and  $r$  denotes a specific relation.
A side point to note is that a subgraph has a single type of edge (a homogeneous graph), and therefore  $e_{ij}^{l}$  is the same within a Graph Transformer corresponding to relation  $r$ . However,  $e_{ij}^{l}$  gets updated over the layers similar to node representation  $h_{i}^{l}$ . We think that it stores a semantic representation of a relation which helps in improving the predictions as described in Section 5.3.
3.3.2 Relation-Specific Subgraph Representation
In particular, this component takes  $h_{w,r}, r \in G_{sub}$  and computes relation-specific subgraph representation  $g_r^\phi$  as a mean of  $h_{w,r}$  using following equation:
$$
g _ {r, M} ^ {\phi} = \frac {\sum_ {w \in \mathcal {A} _ {M}} h _ {w , r}}{| | \mathcal {A} _ {M} | |}, \forall r \in G _ {s u b} \tag {6}
$$
where  $||\mathcal{A}_M||$  denotes the length of the sentence or the number of nodes in a subgraph. Similarly, we create  $g_{r,S}^{\phi}$  for the subgraphs associated with student textual sentence.
3.4 Graph Matching Layer
After obtaining all the subgraph representations which have syntactically and semantically rich information, we utilize the multi-perspective cosine
distance (Wang et al., 2017) to compare  $g_{r,M}^{\phi}$  and  $g_{r,S}^{\phi}$
$$
\mathcal {D} _ {r, k} = \operatorname {c o s i n e} \left(w _ {k} ^ {\cos} \odot g _ {r, M} ^ {\phi}, w _ {k} ^ {\cos} \odot g _ {r, S} ^ {\phi}\right) \tag {7}
$$
$$
\mathcal {D} = \left[ \mathcal {D}, \mathcal {D} _ {r, k} \right] \tag {8}
$$
Where  $k \in \{1, 2, \dots, P\}$  ( $P$  is number of perspectives).  $w_{k}^{cos}$  is a parameter vector, which assigns different weights to different perspectives. With  $P$  perspectives  $d_{1}, d_{2}, \ldots, d_{P}$ , the  $\mathcal{D}_{r,k}$  is updated to  $P$  size. The concatenation of two vectors is denoted using  $[., .]$ , where  $\mathcal{D}$  is initialized with a Null value and later it stores the concatenated value of all  $\mathcal{D}_{r,k}$ .  $\mathcal{D}$  stores the matching score for all relation-specific subgraphs.
3.5 Score Prediction Layer
Student score Score is calculated by using a fully connected layer  $FFN$  which takes  $\mathcal{D}$  as input and has an output layer of a single dimension.
$$
\operatorname {S c o r e} = F F N (\mathcal {D}) \tag {9}
$$
During training phase we have used RMSE loss, where  $y_{i}$  and  $\hat{y}_i$  represents the ground truth and predicted values respectively.
$$
R M S E = \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {N} \left(y _ {i} - \hat {y} _ {i}\right) ^ {2}} \tag {10}
$$
In our experiments, we use the Mohler's dataset (Mohler et al., 2011). It consists of 80 questions of an undergraduate Data Structures course. 2273 student responses are recorded in the dataset, which is evaluated independently by two academicians. We have considered the average scores as model scores. The score lies within a range of 0 to 5. We have considered it as a regression problem.
4.2 Data Processing
As described in section 3.1, we use the Model API from the amrlib library to create Abstract Meaning Representation graphs  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  of a given input sentence  $S$ .
It is crucial to note that we consider the AMR representation as undirected while constructing the adjacency list. This can be intuitively justified as if there exists a relation (say  $A_0$ ) between  $w_{a}$  and
$w_{b}$ $(w_{a}\to w_{b})$  the same relation justifies  $w_{b}\rightarrow$ $w_{a}$  , and could thus be helpful in the final score prediction.
When we parse an original sentence  $S = \{w_{1}, w_{2}, \ldots, w_{n}\}$ , we get a directed AMR graph. Our next step is to convert the AMR to a NetworkX $^4$  graph. While creating the NetworkX graphs, the GloVe $^5$  embeddings for all the words i.e. the nodes in the AMR graph, are embedded as features in the NetworkX graph. We apply Principal Component Analysis (PCA) (Abdi and Williams, 2010) on the original 300D Glove embeddings to reduce it to a lower dimension. Before feeding into the graph transformer, we convert all the NetworkX graphs to DGL format $^6$ . A similar procedure is repeated for all subgraphs.
It is noteworthy that in some cases, AMR representation contains certain phrases like have degree, which is actually a combination of two or more words (have and degree). Such phrases/words don't have a GloVe representation and are thus treated as out-of-vocabulary.
4.3 Parameter Settings
The graph transformer has 2 layers since it gives the best results as observed in preliminary experiments. We use 4 attention heads as we observed that the model performance deteriorates if more/fewer heads are used as described in Section 5.4. We have also added self-loops to include each graph node while updating its representation. The subgraph representation is the mean of node representations. We use  $P = 16$  in the graph matching layer, where  $P$  is the number of perspectives defined in Section 3.4.
We employ the RMSProp optimizer to minimize RMSE loss. The batch size is set to 128 and the initial learning rate to 0.0007. The 'ReduceLROn-Plateau' scheduler is used to reduce the learning rate by a factor of 0.5 when the loss stagnates, with a patience level of 15 epochs. Our implementation uses PyTorch<sup>7</sup>, a popular deep learning framework in Python. All experiments are run on Intel Xenon CPU with 1 Nvidia Quadro P5000 GPU.
4.4Baselines and Metrics
For evaluating MitiGaTe on (Mohler et al., 2011) dataset, we compare against the following base
4https://github.com/networkx/networkxx
5https://bit.ly/glove300D
<https://www.dgl.ai/
<sup>7</sup>https://pytorch.org/
lines. BOW (Mohler et al., 2011) is a simple model based on Bag Of Words. Tfidf (Mohler et al., 2011) is a simple tfidf similarity between  $\mathcal{A}^M$  and  $\mathcal{A}^S$ . Sultan et al. (Sultan et al., 2016), a fast, simple and high performance system which uses Random Forest classifier. Kumar et al. (Kumar et al., 2017) uses a Siamese LSTM network. Word2Vec, GloVe and FastText (Gaddipati et al., 2020) are context independent token embedding models. ELMO, GPT, BERT and GPT-2 (Gaddipati et al., 2020) are deep learning based context based token embedding models. GCN (Kipf and Welling, 2016) performs homogeneous graph convolutions. GAT (Hamilton et al., 2017) performs the attentive weighted sum to update node representation. GraphSAGE (Hamilton et al., 2017) is a framework for inductive representation learning. All GNN baselines use the default subgraph as input, and have 2 layers. RGCN (Schlichtkrull et al., 2018) employs relation specific transformation matrix to incorporate relations in the graph.