SlowGuess/NAACL_2022_OCR huggingface dataset:
We use Root mean square error (RMSE) for performance evaluation, which gives a fair assessment of students' responses. A lower metric value corresponds to better model.
In this section, we attempt to answer following questions: RQ1. How does each subgraph influence the final results? (Section 5.2) RQ2. Does incorporating edge representation while computing node representation improve the final results? (Section 5.3) RQ3. Can MitiGaTe provide feedback to students i.e. Why did a student lose marks? (Section 5.5)
5.1 Results on Mohler's Data
Table 1 presents the results of our model on the Mohler's dataset. We can see that our model outperforms all of the previous models by a significant margin. It demonstrates the importance of incorporating relation-enriched structural context in the tokens for effective text comparison. The existing baseline models can be categorized as (i) Handcrafted features (ii) Deep Learning-based models (iii) Word Embeddings based on sequential context information (iv) Graph-based, which store the structural information and relationship information.
ELMo outperforms the other transfer learning models. It is fundamentally a direct result of the capacity of the model to assign context-dependent
Model	Features	RMSE
BOW	SVM Rank	1.042
SVR	0.999
tfidf	SVR	1.022
Sultan	LR + SIM	0.887
Kumar	Siamese	0.830
Word2Vec	SOWE	1.025
SIM+Verb	1.016
GloVe	SOWE	1.036
SIM+Verb	1.002
FastText	SOWE	1.023
SIM+Verb	0.956
ELMO	SOWE	0.978
GPT	SOWE	1.082
BERT	SOWE	1.057
GPT-2	SOWE	1.065
GCN	HG	0.991
GAT	AWG	0.974
GraphSAGE	HG	0.986
RGCN	HtG	0.892
MitiGaTe	MRGT	0.762
Table 1: MitiGaTe Evaluation on Mohler Dataset (Mohler et al., 2011). SVM = Support Vector Machine, SVR = Support Vector Regression; LR = Length Ratio between desired answer and student answer; SIM = Similarity score; SOWE = Sum Of Word Embeddings; HG = Homogeneous Graph; AWG = Attention Weighted Graph, HtG = Heterogeneous Graph, MRGT = Multi-relational Graph Transformer.
word-vectors. RGCN performs better than other graph-based baselines because it incorporates the relation-specific information in the form of a heterogeneous graph. Nevertheless, it focuses only on the characteristics of the relations and does not study different types of features associated with a node. Our results establish that incorporating the relation-enriched structural information (MitiGaTe) contributes to significant performance improvement in the downstream task. This observation is generic and can be applied to different applications beyond ASAG.
5.2 Influence of Subgraphs
In this section, we investigate how each subgraph influences the final results of our best model Mit-iGaTe. Table 2 shows the effect of using different combinations of relation-specific subgraphs on the result. Using only the default subgraph implies that the model does not consider the relational
Model	RMSE
MitiGaTe	0.762
Only default	0.968
Only A0	1.017
Only A1	1.020
Only other	1.144
Only default + other	1.178
Only default + A1	0.949
Only default + A0	0.872
MitiGaTe - A0	0.925
MitiGaTe - default	0.888
MitiGaTe - A1	0.816
MitiGaTe - other	0.794
information in inputs, i.e., considers a simple homogeneous graph. We see that it performs better than GCN (mentioned in Table 1) because it uses a Graph Transformer encoder. On using  $A_0$ ,  $A_1$  and other subgraphs separately, the model performance degrades as they capture a subset of the relations captured by default.
Using the default subgraph along with the primary relations  $A_0$  and  $A_1$  improves the performance because incorporating multiple relations supplements the syntactic and semantic information. We think that the reason for performance degradation to 1.178 RMSE on using the other subgraph along with default, is that we have stored all remaining relations available in the AMR under the other class.
Furthermore we can observe that on removing the individual subgraphs one-by-one from MitiGaTe, the performance deteriorates in all cases. These results corroborate the hypothesis that utilizing multi-relational information helps in improving the overall outcomes. The relation  $A_0$  stores the information related to agents or cause's, and therefore it influences the results the most.
5.3 Influence of Edge Representations
Table 5.3 demonstrates that incorporating the edge representations in the graph transformer certainly helps in improving the attention weights, and therefore the overall results have improved significantly. The edge embeddings are initialized with random values, but they get updated in the layers of the graph transformer. As stated earlier, we expect
that the edge embeddings store a relation's semantic representation. From the results, we can infer that a relation's semantic representation plays an essential role in the overall process.
Table 2: Influence of relation-specific subgraphs on performance. MitiGaTe uses default  $+ {A}_{0} + {A}_{1} +$  other subgraphs.
Model	RMSE
MitiGaTe w/o edge representation	0.864
MitiGaTe w/ edge representation	0.762
5.4 Analysis of Parameters
We study the impact of the number of transformer layers and attention heads on MitiGaTe. The results are summarized in Figure 4. We vary the number of layers keeping the number of heads fixed as 4. The performance first improves with increasing layers as a deeper model receives better information from multi-hop neighbors. However, too many layers lead to performance degradation, and we see that this is due to the over-smoothing problem discussed by (Li et al., 2018). Next, the number of attention heads is varied keeping the number of layers fixed as 2. We observe that more attention heads improve the performance during training but are redundant during the testing. This is consistent with the observation of (Michel et al., 2019).
Figure 4: Effect of parameters on RMSE (Layers, Heads).
5.5 Case Study: Feedback to Students
In addition to scoring a student response, a significant focus of the human evaluation is giving feedback on why a student lost marks. MitiGaTe matches tokens at the relation-level as illustrated in Figure 5. Student C scored  $3/5$  because of the corresponding matching of words belonging to the same relation in student and model answers. There exists the same relation  $A_{1}$  between words refine and solve, in model answer and words refine and solution in student answer. Similarly the other and  $A_{0}$  relations have been highlighted in the Figure 5. However, the student loses marks because there are a few relationships between words like code and
Figure 5: Example from Figure 1, demonstrating how the model performs matching at the relation-level. The green, red and blue color edges indicate  $A_0$ ,  $A_1$ , other relations respectively. Dotted lines denote the relation-specific token matching. Grey nodes indicate the missing content, and pink nodes denote the extra information provided in student answer.
phase in the model answer, which are not present in the student response. These relations can be used to highlight the (in)correct parts of the student answer. We see that the relational matching information intuitively acts as feedback to explain the final scores.
In this paper, we have proposed MitiGaTe for ASAG. It prepares token embeddings considering the structural context of a sentence and thus provides a more efficient matching method by considering multiple relations at a granular level. Experimental results show that MitiGaTe outperforms the existing ASAG systems by a significant margin, and can be extended to give an intuitive feedback to explain the provided score.
In the future, we would like to investigate how to deal with long and multi-lingual answers. Our approach uses an AMR graph, and thus such tasks will need a compatible AMR parser. We also aim to incorporate the explainability of the final scoring to generate more comprehensive evaluator feedback.
Hervé Abdi and Lynne J Williams. 2010. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433-459.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pages 81-91.
Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. 2016. Automatic text scoring using neural networks. arXiv preprint arXiv:1606.04289.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Jasmijn Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima'an. 2017. Graph convolutional encoders for syntax-aware neural machine translation. arXiv preprint arXiv:1704.04675.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364.
Danilo Croce, Alessandro Moschitti, and Roberto Basili. 2011. Structured lexical similarity via convolution kernels on dependency trees. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1034-1046.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29:3844-3852.
Vijay Prakash Dwivedi and Xavier Bresson. 2020. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699.
Myroslava O Dzikovska, Rodney D Nielsen, Chris Brew, Claudia Leacock, Danilo Giampiccolo, Luisa Bentivogli, Peter Clark, Ido Dagan, and Hoa T Dang. 2013. Semeval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge. Technical report, NORTH TEXAS STATE UNIV DENTON.
Sasi Kiran Gaddipati, Deebul Nair, and Paul G Plöger. 2020. Comparative evaluation of pretrained transfer learning models on automatic short answer grading. arXiv preprint arXiv:2009.01303.
Wael H Gomaa, Aly A Fahmy, et al. 2013. A survey of text similarity approaches. international journal of Computer Applications, 68(13):13-18.
William L Hamilton, Rex Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 1025-1035.
Sarah Hassan, Aly A. Fahmy, and Mohammad El-Ramly. 2018. Automatic short answer scoring based on paragraph embeddings. International Journal of Advanced Computer Science and Applications, 9(10).
Hua He, Kevin Gimpel, and Jimmy Lin. 2015. Multiperspective sentence similarity modeling with convolutional neural networks. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1576-1586.
Michael Heilman and Nitin Madnani. 2013. Ets: Domain adaptation and stacking for short answer scoring. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 275-279.
Lianzhe Huang, Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2019. Text level graph neural network for text classification. arXiv preprint arXiv:1910.02356.
Sergio Jimenez, Claudia Becerra, and Alexander Gelbukh. 2013. Softcardinality: Hierarchical text overlap for student response analysis. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 280-284.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Sachin Kumar, Soumen Chakrabarti, and Shourya Roy. 2017. Earth mover's distance pooling over siamese lstms for automatic short answer grading. In *IJCAI*, pages 2046-2052.
Yuquan Le, Zhi-Jie Wang, Zhe Quan, Jiawei He, and Bin Yao. 2018. Acv-tree: A new method for sentence similarity modeling. In *IJCAI*, pages 4137-4143.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI conference on artificial intelligence.
Xien Liu, Xinxin You, Xiao Zhang, Ji Wu, and Ping Lv. 2020. Tensor graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8409-8416.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. arXiv preprint arXiv:1703.04826.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? Advances in Neural Information Processing Systems, 32:14014-14024.
Michael Mohler, Razvan Bunescu, and Rada Mihalcea. 2011. Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 752-762, Portland, Oregon, USA. Association for Computational Linguistics.
Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similarity. In Proceedings of the AAAI conference on artificial intelligence, volume 30.
Niels Ott, Ramon Ziai, Michael Hahn, and Detmar Meurers. 2013. Comet: Integrating different levels of linguistic modeling for meaning assessment. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 608-616.
Brian Riordan, Andrea Horbach, Aoife Cahill, Torsten Zesch, and Chungmin Lee. 2017. Investigating neural architectures for short answer scoring. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 159-168.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European semantic web conference, pages 593-607. Springer.
Aliaksei Severyn, Massimo Nicosia, and Alessandro Moschitti. 2013. Building structures from classifiers for passage reranking. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 969-978.
Md Arafat Sultan, Cristobal Salazar, and Tamara Sumner. 2016. Fast and easy short answer grading with high accuracy. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1070-1075.
Shikhar Vashisth, Manik Bhandari, Prateek Yadav, Piyush Rai, Chiranjib Bhattacharyya, and Partha Talukdar. 2019. Incorporating syntactic and semantic information in word embeddings using graph convolutional networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3308-3318.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. arXiv preprint arXiv:1702.03814.
Weiwen Xu, Huihui Zhang, Deng Cai, and Wai Lam. 2021. Dynamic semantic graph construction and reasoning for explainable multi-hop science question answering. arXiv preprint arXiv:2105.11776.
Xi Yang, Yuwei Huang, Fuzhen Zhuang, Lishan Zhang, and Shengquan Yu. 2018. Automatic chinese short answer grading with deep autoencoder. In International Conference on Artificial Intelligence in Education, pages 399-404. Springer.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7370-7377.
Siyuan Zhao, Yaqiong Zhang, Xiaolu Xiong, Anthony Botelho, and Neil Heffernan. 2017. A memory-augmented neural model for automated grading. In Proceedings of the fourth (2017) ACM conference on learning@ scale, pages 189-192.
A AMR Representation
In this section, we demonstrate the generated AMR graphs of the model and student answers shown in Figure 1.
Figure 6: Student B: Refining the solution
Figure 7: Student A: Directly: Refining, coding. Because Refining is right before the Testing Phase and Coding is right after the Testing Phase. Indirectly: Production, Maintenance.
Figure 8: Student D: All stages are influenced except setting the program requirements. If a test fails, it can change the whole design, implementation, etc of a program.
Figure 9: Student C: Refining the solution, Production and Maintenance are all influenced by the Testing stage.
Figure 10: Model Answer: The testing stage can influence both the coding stage (phase 5) and the solution refinement stage (phase 7)