SlowGuess/ACL_2020_OCR huggingface dataset:
evance layer. With the help of t-SNE (Maaten and Hinton, 2008), the visualization result was shown in Figure 3. Subfigure 3a tells the true response distribution of one prompt, "describe a special meal that you have had, what the meal was, who you had this meal with and explain why this meal was special", which has a clear-semantic topic "meal". Meanwhile, subfigure 3b reveals the response distribution using our semantic matching representation on the same prompt as subfigure 3a.
We can see that semantic matching representation of our model maintains good performance on this kind of prompt, which has one clear-semantic topic to limit the discussion in one scope. Additionally, some prompts are open to discuss, which are divergent. Given a case of the prompt "what do you do in your spare time", and we can observe its true response distribution in subfigure 3c. Compared with it in subfigure 3c, our model tends to predict responses on-topic, seen in subfigure 3d, because high on-topic recall (0.999) is limited.
4.4 Negative Sampling Augmentation Method
To investigate the impact of training data size, we conduct some experiments with varying sizes of training data. In figure 4, we find that the larger the training data size, the better the performance.
Table 4: The comparison of different models based on over 0.999 on-topic recall on seen and unseen benchmarks. AOR means Average Off-topic Recall (%) and PRR3 means Prompt Ratio over off-topic Recall 0.3 (%).
Model	Seen	Unseen
PPR3	AOR	PPR3	AOR
GCBiA	93.6	79.2	68.0	45.0
+ neg sampling	94.2	88.2	79.4	69.1
Table 5: The performance of GCBiA with negative sampling augmentation method conditioned on over 0.999 on-topic recall.
To augment training data and strengthen the generalization of the off-topic response detection
(a) True resp distribution (b) Model's resp distribu-on clear-semantic topic tion on clear-semantic topic prompt. prompt.
(c) True response distribu-(d) Model's resp distribution on divergent prompt. tion on divergent prompt.
Figure 3: The analysis of response distribution on different types of prompts. The yellow and black colours represent the on-topic and off-topic response results respectively.
Figure 4: Trends of AOR (Average Off-topic Recall) on seen and unseen prompts with datasize variation.
model for unseen prompts, we proposed a new and effective negative sampling method for off-topic response detection task. Comparing with the previous method of generating only one negative sample for each positive one, we generated two. The first one is chosen randomly as before, and the second one consists of words shuffled from the first one. This method contributes to the diversity of negative samples of training data. The size of our training data reaches 1.67M, compared with 1.12M in the previous negative sampling method. To make training data balanced, we gave the weight of positive and negative samples: 1 and 0.5, respectively. As is shown in Table 5, a significant performance improvement (+9.0 seen AOR and +24.1
unseen AOR) is achieved by this negative sampling method. Our model GCBiA equipped with negative sampling augmentation can achieve  $88.2\%$  and  $69.1\%$  average off-topic response recall on seen and unseen prompts, conditioned on 0.999 on-topic recall.
In this paper, we conducted a series of work around the task of off-topic response detection. First of all, a model framework of five major layers was proposed, within which bi-attention mechanism and convolutions were used to well capture the topic words of prompts and key-phrase of responses, and gated unit as relevance layer was applied to better obtaining semantic matching representation, as well as residual connections with each major layer. Moreover, the visualization analysis of the off-topic model was given to study the essence of the model. Finally, a novel negative sampling augmentation method was introduced to augment off-topic training data. We verified the effectiveness of our approach and achieved significant improvements on both seen and unseen test data.
We are grateful to our colleague Bin Wang for helping with the ASR system. We thank our colleague Puyu Chen for proofreading. Last but not least, we thank the anonymous reviewers for their invaluable comments.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettle-moyer. 2018. Quac: Question answering in context. arXiv preprint arXiv:1808.07036.
Timothy Dozat. 2016. Incorporating nesterov momentum into adam.
Keelan Evanini and Xinhao Wang. 2014. Automatic detection of plagiarized spoken responses. In Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications, pages 22-27.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Derrick Higgins and Michael Heilman. 2014. Managing what we can measure: Quantifying the susceptibility of automated scoring systems to gaming behavior. Educational Measurement: Issues and Practice, 33(3):36-46.
Chong Min Lee, Su-Youn Yoon, Xihao Wang, Matthew Mulholland, Ikkyu Choi, and Keelan Evanini. 2017. Off-topic spoken response detection using siamese convolutional neural networks. In INTERSPEECH, pages 1427-1431.
Karen E Lochbaum, Mark Rosenstein, PW Foltz, Marcia A Derr, et al. 2013. Detection of gaming in automated scoring of essays with the iaea. In National Council on Measurement in Education Conference (NCME), San Francisco, CA.
Annie Louis and Derrick Higgins. 2010. Off-topic essay detection using short prompt texts. In proceedings of the NAACL HLT 2010 fifth workshop on innovative use of NLP for building educational applications, pages 92-95. Association for Computational Linguistics.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
Andrey Malinin, Kate Knill, Anton Ragni, Yu Wang, and Mark JF Gales. 2017. An attention based model for off-topic spontaneous spoken response detection: An initial study. In *SLaTE*, pages 144-149.
Andrey Malinin, Rogier C Van Dalen, Yu Wang, Katherine Mary Knill, and Mark John Gales. 2016. Off-topic response detection for spontaneous spoken english assessment.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi, and Sanjeev Khudanpur. 2018. Semi-orthogonal low-rank matrix factorization for deep neural networks. In *Interspeech*, pages 3743–3747.
Daniel Povey, Vijayaditya Peddinti, Daniel Galvez, Pegah Ghahremani, Vimal Manohar, Xingyu Na, Yiming Wang, and Sanjeev Khudanpur. 2016. Purely sequence-trained neural networks for asr based on lattice-free mmi. In *Interspeech*, pages 2751-2755.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603.
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 189-198.
Xinhao Wang, Su-Youn Yoon, Keelan Evanini, Klaus Zechner, and Yao Qian. 2019. Automatic detection of off-topic spoken responses using very deep convolutional neural networks. Proc. Interspeech 2019, pages 4200-4204.
Wenpeng Yin, Katharina Kann, Mo Yu, and Hinrich Schütze. 2017. Comparative study of cnn and rnN for natural language processing. arXiv preprint arXiv:1702.01923.
Su-Youn Yoon and Shasha Xie. 2014. Similarity-based non-scorable response detection for automated speech scoring. In Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications, pages 116-123.
Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018. Recent trends in deep learning based natural language processing. *IEEE Computational intelligence* magazine, 13(3):55-75.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. *Qanet: Combining local convolution with global self-attention for reading comprehension*. arXiv preprint arXiv:1804.09541.