SlowGuess/ACL_2020_OCR huggingface dataset:
Semantic Evaluation (SemEval@ACL 2017), pages 27-48, Vancouver, Canada. Association for Computational Linguistics.
Preslav Nakov, Lluis Marquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhakim Freihat, James Glass, and Bilal Randeree. 2016. SemEval-2016 Task 3: Community Question Answering. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval@NAACL-HLT 2016), pages 525-545, San Diego, USA. Association for Computational Linguistics.
Preslav Nakov, Lluis Marquez, Magdy Walid, Alessandro Moschitti, James Glass, and Bilal Randeree. 2015. SemEval-2015 task 3: Answer Selection in Community Question Answering. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval@NAACL-HLT 2015), pages 269–281, Denver, USA. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Malte Ostendorff, Peter Bourgonje, Maria Berger, Julian Moreno-Schneider, Georg Rehm, and Bela Gipp. 2019. Enriching BERT with Knowledge Graph Embeddings for Document Classification. In Proceedings of the 15th Conference on Natural Language Processing (KONVENS), Erlangen, Germany.
Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text Matching as Image Recognition. In Proceedings of the Thirtieth Conference on Artificial Intelligence (AAAI), pages 2793-2799, Phoenix, USA. AAAI Press.
Nicole Peinelt, Maria Liakata, and Dong Nguyen. 2019. Aiming beyond the Obvious: Identifying Non-Obvious Cases in Semantic Similarity Datasets. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2792-2798, Florence, Italy. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2227-2237, New Orleans, USA. Association for Computational Linguistics.
Zengchang Qin, Marcus Thint, and Zhiheng Huang. 2009. Ranking Answers by Hierarchical Topic Models. In Next-Generation Applied Intelligence, 22nd
International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems (IEA/AIE 2009), volume 5579, pages 103-112, Tainan, Taiwan. Springer.
Chuanqi Tan, Furu Wei, Wenhui Wang, Weifeng Lv, and Ming Zhou. 2018. Multiway Attention Networks for Modeling Sentence Pairs. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI), pages 4411-4417, Stockholm, Sweden.
Gaurav Singh Tomar, Thyago Duque, Oscar Täckström, Jakob Uszkoreit, and Dipanjan Das. 2017. Neural Paraphrase Identification of Questions with Noisy Pretraining. In Proceedings of the First Workshop on Subword and Character Level Models in NLP, pages 142-147, Copenhagen, Denmark. Association for Computational Linguistics.
Quan Hung Tran, Vu Tran, Tu Vu, Minh Nguyen, and Son Bao Pham. 2015. JAIST: Combining Multiple Features for Answer Selection in Community Question Answering. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 215-219, Denver, USA. Association for Computational Linguistics.
Li Wang, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du. 2018. A Reinforced Topic-Aware Convolutional Sequence-to-Sequence Model for Abstractive Text Summarization. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI), pages 4453–4460, Stockholm, Sweden.
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral Multi-Perspective Matching for Natural Language Sentences. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI), pages 4144-4150, Melbourne, Australia.
Guoshun Wu, Yixuan Sheng, Man Lan, and Yuanbin Wu. 2017. ECNU at SemEval-2017 Task 3- Using Traditional and Deep Learning Methods to Address Community Question Answering Task. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval@ACL 2017), pages 356-360, Vancouver, Canada. Association for Computational Linguistics.
Wen-tau Yih, Kristina Toutanova, John C Platt, and Christopher Meek. 2011. Learning Discriminative Projections for Text Similarity Measures. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning (CoNLL 2011), pages 247-256, Portland, USA. Association for Computational Linguistics.
Jianhua Yin and Jianyong Wang. 2014. A Dirichlet Multinomial Mixture Model-based Approach for Short Text Clustering. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '14, pages 233-242, New York, USA. ACM Press.
7053
Dataset	Sentence pair	L
MSRP	There are only 2,000 Roman Catholics living in Banja Luka now. There are just a handful of Catholics left in Banja Luka.	1
Quora	Which is the best way to learn coding? How do you learn to program?	1
SemEval A	Anybody recommend a good dentist in Doha? Dr Sarah Dental Clinic	1
SemEval B	Where I can buy good oil for massage? Blackheads - Any suggestions on how to get rid of them??	0
SemEval C	Can anybody tell me where is Doha clinic? Dr. Rizwi - Al Ahli Hospital	0
Table 6: Examples from different datasets. Labels (L) indicate if the second sentence is a paraphrase (for paraphrasing tasks) or relevant (for QA tasks).
B LDA Topic Examples
T1: life purpose important thing real biggest
T2: drink water coffee tea drinking good
T3:Pokémon flight car ticket train fly
T4: school university college high students student
T5: chemical determine formula acid determined san
Table 7: Top key words for example topics learned by an LDA model with 90 topics on the Quora training set.
T1:	regiment cavalry north 3rd passenger fort
T2:	court judge federal district supreme file
T3:	windows server software microsoft 2003 system
T4:	president bush time presidential report george
T5:	hospital condition study risk cancer women
Table 8: Top key words for example topics learned by an LDA model with 80 topics on the MSRP training set.
T1:	gym club pool fitness gyms swimming
T2:	drink good club music night alcohol
T3:	husband sponsorship wife company sponsor work
T4:	day Eid holidays days kadan hours
T5:	time doha bus area morning early
C GSDMM Topic Examples
Table 9: Top key words for example topics learned by an LDA model with 70 topics on the training set of all three SemEval tasks combined.
T1:	difference examples law social science
T2:	effects earthquake major compare cambodia
T3:	arbitration court cards australia world
T4:	panel solar provider installation california
T5:	get best rid skin remove
Table 10: Top key words for example topics learned by a GSDMM model with 90 topics on the Quora training set.
T1:	cases said number year reported sales meeting
T2:	states united wrong sense deal
T3:	two killed united states people government
T4:	condition hospital center taken medical county
T5:	charges commission arrested exchange
Table 11: Top key words for example topics learned by a GSDMM model with 80 topics on the MSRP training set.
T1:	know qatar years many indian qatari
T2:	good qatar live doha know dog
T3:	arabic doha best people time english
T4:	month like 000 car compound villa
T5:	time find visa working company study
D Hyper-Parameters for Topic-Aware Models
Topic model hyper-parameters were chosen based on development set F1 scores of the topic baseline. We tried number of topics: 10, 20, 30, 40, 50, 60, 70, 80, 90, 100 and alpha values: 0.1, 1, 10, 50. The topic baselines and tBERT models use topic models with the same hyper-parameters as listed in Table 13.
Table 12: Top key words for example topics learned by a GSDMM model with 70 topics on the training set of all three SemEval tasks combined.
MSRP	Quora	SemEval
A	B	C
# of topics	80	90	70	80	70
LDA alpha values	1	1	50	10	10
GSDMM alpha values	0.1	0.1	0.1	0.1	0.1
Table 13: Tuned topic model hyper-parameters.
7054
E Hyper-Parameters for BERT-Based Models
Table 14 reports additional hyper-parameters for BERT and tBERT. The learning rate was tuned based on development set F1 score per seed and model using grid search (2e-5, 3e-5 or 5e-5).
MSRP	Quora	SemEval
A	B	C
batch size	32	32	16	32	16
BERT
lrate (1st seed)	5e-5	2e-5	3e-5	2e-5	2e-5
lrate (2nd seed)	5e-5	2e-5	2e-5	2e-5	3e-5
tBERT
lrate (1st seed)	3e-5	3e-5	2e-5	2e-5	3e-5
lrate (2nd seed)	5e-5	2e-5	2e-5	3e-5	2e-5
F Training time
Table 14: Tuned hyper-parameters for BERT-based models. lrate = learning rate.
MSRP	Quora	SemEval
A	B	C
BERT
3 epochs	13	839	223	26	340
9 epochs	44	2710	638	75	1047
tBERT
3 epochs	13	885	211	24	348
9 epochs	42	2916	658	75	1082
Table 15: Average training time on one NVIDIA Tesla K80 GPU in minutes.
G Longer Finetuning Experiment
Longer BERT finetuning does not surpass tBERT's best performance from the first 3 epochs (dotted line) while considerably increasing training time (compare Appendix F).
Figure 3: Performance of BERT and tBERT on development set when trained for up to 9 epochs. The dotted line indicates tBERT's best performance within the first 3 epochs.
7055