SlowGuess/ACL_2020_OCR huggingface dataset:
Table 2 presents evaluation metrics for the three baselines against three variants of our sequence labelling model. The first variant does not use pre-trained embeddings from language models, the second uses just pre-trained ELMo, while the third combines pre-trained embeddings from multiple language models using CDME. All three variants use fastText for word embeddings (refer Section 6.1), character-based embeddings, and do not have attention mechanism before the final CRF layer (refer Section 6.2).
As one would expect, the Context Baseline performs much better than the Sentence Only Baseline. The sequence labelling models, however, outperform both the baselines by a huge margin, demonstrating the effectiveness of the model on the task. Specifically, the best performance is achieved by combining pre-trained embeddings from multiple language models trained on different data sources. It significantly outperforms the model using embeddings from a single pre-trained model on English (explored in Section 6.3).
In the following section we present results from the various model components we explored.
6.1 Effect of fastText
Row 1 and 4 in Table 3 presents the comparison between models using word embeddings from
<sup>4</sup>https://www/elastic.co/products/elasticsearch
Model	P	R	F1
Sent. Only Baseline	47.77	31.75	38.15
Sent. Context Baseline	52.52	34.03	41.3
Supervised Text Segmentation Baseline	44.13	40.43	42.20
SL w/o LM embeddings	74.57	75.51	75.04
SL + pre-trained ELMo	76.88	74.49	75.67
SL + CDME combined pre-trained Embeddings	78.30	79.29	78.80
Table 2: Results comparing the three baselines against variants of our sequence labelling model. The best performing variant uses CDME to combine pre-trained embeddings from multiple language models trained on different datasources.
Model	P	R	F1
Word2Vec (w/o Attn)	65.20	58.59	61.72
+ weighted Attn.	62.34	57.0	59.55
+ un-weighted Attn.	69.21	56.15	62.0
fastText	74.57	75.51	75.04
Table 3: Results for experiments between using Word2Vec and fastText embeddings. Also includes results of using attention on top of the model with Word2Vec. Since attention results were not promising, we did not repeat them with fastText.
word2vec andFAST. Both word2vec and fastText embeddings are trained on all posts in the Ask Ubuntu dataset. As we can see, FAST gives a marked improvement over using embeddings from word2vec. This is probably due to the nature of the vocabulary in our task. Since large portions of questions are spans of command output or error messages a lot of tokens appear very rarely. In fact, out of the 62,501 unique tokens in the dataset,  $57\%$  appear just once, and  $78\%$  appear 3 or fewer times. However, the characters in these tokens are probably very informative (for example "http" in a token would signal that the token is a URL). Therefore, FAST, which uses n-grams from a token to compute embeddings, would emit more meaningful representations.
As a simple experiment, we check the similarity of two URLs from the dataset that appear just once - http://paste.ubuntu.com/1403448/ and http://paste.ubuntu.com/14545476/. While the cosine similarity of Word2Vec vectors for the two is  $-0.07$ , the similarity between the fastText vectors is 0.99.
6.2 Effect of Attention
Given the long tickets in our dataset, and unreasonably long lengths of spans for labels like
command output or error messages, we explored the usefulness of attention in our model. We used the Scaled Dot-Product Attention as in (Vaswani et al., 2017). Rows 2 and 3 in Table 3 present the results of using attention. We find that weighted attention actually hurts performance. This could be because of the large number of extra parameters introduced in the calculation of Key, Value, and Query matrices. While the un-weighted version gets around this by using the bi-directional GRU hidden states as all 3 matrices, it doesn't improve results significantly either.
6.3 Effect of Contextual Pre-Trained Embeddings
As detailed in Section 4.3, we explore the impact of pre-trained contextual embeddings. We also test our hypothesis, that combining pre-trained embeddings from different data sources would perform better on our task than using embeddings from a language model trained on a single data source. The combination is also performed in two ways - naive concatenation of embeddings from all language models, and weighted combination using DME and CDME as in Kiela et al. (2018).
Table 4 summarizes these results. For the simple concatenation method, we present results for the best  $n$ -way combination of embeddings from different data sources, for each  $n$  (1, 2, 3, and 4). We find that combining embeddings from multiple language models trained on different data sources considerably outperforms using embeddings from a single pre-trained model (using both the naive concatenation and CDME). This is an artifact of the support problems containing large sections of nonnatural language text. We also find that contextual weighting does better than a simple concatenation.
Model	P	R	F1
No Pretraining	74.57	75.51	75.04
Simple Concat - 1 (en)	76.88	74.49	75.67
Simple Concat - 2 (en + config)	77.67	76.12	76.89
Simple Concat - 3 (en + code + config)	79.64	77.72	78.67
Simple Concat - 4 (ALL)	76.05	76.65	76.35
DME	77.42	75.82	76.61
CDME	78.30	79.29	78.80
Table 4: Results comparing the models using various pre-trained embeddings. The en data source is the downloaded pre-trained ELMo model. For simple concatenation, we present the results for the best model at each n combinations of data sources. For example, when concatenating any 2 datasources, the en + config combination gives the best performance.
Method	MRR
Full Question	0.292
Segmented Question - Gold	0.300
Segmented Question - Predicted	0.298
Table 5: Retrieval results, comparing the performance of querying with the full question against segmented question (gold segments and predicted segments)
6.4 Retrieval of the Correct Answer
Table 5 presents results for the retrieval experiment. We show that weighing identified segments of the question with separate weights improves retrieval of the correct answer over a query with all tokens from the question. We also present results from the gold annotations of segments for these questions, as an upper-bound of the performance improvement we can hope to achieve.
In this paper, we introduce and address an important problem towards a better understanding of support tickets - segmentation of various nonnatural language segments. We create an annotated dataset for the task, on questions from the publicly available website, Ask Ubuntu. We also study the performance of the most recent Recurrent Neural Network-based approaches to sequence labelling, on this task. In the end, we propose the novel idea of combining pre-trained embeddings from language models trained on different data sources, which substantially improves performance. We
also demonstrate the usefulness of the task with improvements in retrieval of the correct answer. Our future research direction includes a thorough study of differences in this dataset with actual tickets, and potential for transfer. It is still valuable to study models on open datasets, however, as these are readily available to the community.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638-1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Jason P.C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4:357-370.
Freddy Y. Y. Choi. 2000. Advances in domain independent linear text segmentation. In Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, NAACL 2000, page 2633, USA. Association for Computational Linguistics.
Grzegorz Chrupa. 2013. Text segmentation with character-level text embeddings. Workshop on Deep Learning for Audio, Speech and Language Processing, ICML 2013, Atlanta, United States.
Abhirut Gupta, Anupama Ray, Gargi Dasgupta, Gautam Singh, Pooja Aggarwal, and Prateeti Mohapatra. 2018. Semantic parsing for technical support questions. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3251-3259, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Shilin He, Jieming Zhu, Pinjia He, and Michael R Lyu. 2016. Experience report: system log analysis for anomaly detection. In 2016 IEEE 27th International Symposium on Software Reliability Engineering (IS-SRE), pages 207-218.
Marti A. Hearst. 1997. TextTiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics, 23(1):3364.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv e-prints, page arXiv:1508.01991.
Richard Johansson and Alessandro Moschitti. 2010. Syntactic and semantic structure for opinion expression detection. In Proceedings of the Fourteenth
Conference on Computational Natural Language Learning, CoNLL 10, page 6776, USA. Association for Computational Linguistics.
Douwe Kiela, Changhan Wang, and Kyunghyun Cho. 2018. Dynamic meta-embeddings for improved sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1466–1477, Brussels, Belgium. Association for Computational Linguistics.
Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. 2018. Text segmentation as a supervised learning task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 469-473, New Orleans, Louisiana. Association for Computational Linguistics.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270, San Diego, California. Association for Computational Linguistics.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074, Berlin, Germany. Association for Computational Linguistics.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS17, page 62976308, Red Hook, NY, USA. Curran Associates Inc.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS'13, pages 3111-3119, USA. Curran Associates Inc.
Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1756-1765, Vancouver, Canada. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages
2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Marek Rei, Gamal Crichton, and Sampo Pyysalo. 2016. Attending to characters in neural sequence labeling models. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 309-318, Osaka, Japan. The COLING 2016 Organizing Committee.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Kai Wang, Zhao-Yan Ming, Xia Hu, and Tat-Seng Chua. 2010. Segmentation of multi-sentence questions: Towards effective question retrieval in CQA services. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 10, page 387394, New York, NY, USA. Association for Computing Machinery.
Yizhong Wang, Sujian Li, and Jingfeng Yang. 2018. Toward fast and accurate neural discourse segmentation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 962-967, Brussels, Belgium. Association for Computational Linguistics.