SlowGuess/NAACL_2021_OCR huggingface dataset:
We conduct our experiments using PyTorch Huggingface's transformers (Wolf et al., 2019) and the sentence-transformers framework $^{10}$  (Reimers and Gurevych, 2019). The latter showed that BERT outperforms other transformer-like networks when used as bi-encoder. For English datasets, we use bert-base-uncased and for the Spanish dataset we
<sup>9</sup>https://github.com/wasiahmad/paraphrase_identification
$^{10}$ https://github.com/UKPLab/sentence-transformers
$^{7}$ https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs  
 $^{8}$ https://drive.google.com/file/d/0B0PTAo-BnaQWlsZ19FZ311c28
Task	Regression (ρ × 100)	Classification (F1)
Model / Dataset	(Seed Opt.)	Spanish-STS	BWS (cross-topic)	BWS (in-topic)	Quora-QP	MRPC
Baseline	-	30.27	5.53	6.98	66.67	80.80
USE (Yang et al., 2019)	-	86.86	53.43	57.23	74.16	81.51
BERT	X	77.50 ± 1.49	65.06 ± 1.06	65.91 ± 1.20	80.40 ± 1.05	88.95 ± 0.67
SBERT	X	68.36 ± 5.28	58.04 ± 1.46	61.20 ± 1.66	73.44 ± 0.65	84.44 ± 0.68
BERT (Upper-bound)	✓	77.74 ± 1.24	65.78 ± 0.78	66.54 ± 0.94	81.23 ± 0.93	89.00 ± 0.56
SBERT (Lower-bound)	✓	72.07 ± 2.05	60.54 ± 0.99	63.77 ± 2.29	74.66 ± 0.31	84.39 ± 0.51
SBERT-NLPAug	✓	74.11 ± 2.58	58.15 ± 1.66	61.15 ± 0.86	73.08 ± 0.42	84.47 ± 0.79
AugSBERT-R.S.	✓	62.05 ± 2.53	59.95 ± 0.70	64.54 ± 1.90	73.42 ± 0.74	82.28 ± 0.38
AugSBERT-KDE	✓	74.67 ± 1.01	61.49 ± 0.71	69.76 ± 0.50	79.31 ± 0.46	84.33 ± 0.27
AugSBERT-BM25	✓	75.08 ± 1.94	61.48 ± 0.73	68.63 ± 0.79	79.01 ± 0.45	85.46 ± 0.52
AugSBERT-S.S.	✓	74.99 ± 2.30	61.05 ± 1.02	68.06 ± 0.93	77.20 ± 0.41	82.42 ± 0.32
AugSBERT-BM25+S.S.	✓	76.24 ± 1.42	59.41 ± 0.98	63.30 ± 1.34	72.45 ± 0.77	82.68 ± 0.33
Table 5: Summary of all the datasets being used for the in-domain tasks in this paper. STS and BWS are regression tasks, where we report Spearman's rank correlation  $\rho \times 100$ . Quora-QP and MRPC are classification tasks, where we report  $F_{1}$  score of the positive class. Scores with the best AugSBERT strategy are highlighted. Corresponding development set performances can be found in Appendix G, Table 12.
use bert-base-multilingual-cased. Every AugSBERT model exhibits computational speeds identical to the SBERT model (Reimers and Gurevych, 2019).
Cross-encoders We fine-tune the BERT-uncased model by optimizing a variety of hyperparameters: hidden-layer sizes, learning-rates and batch-sizes. We add a linear layer with sigmoid activation on top of the [CLS] token to output scores 0 to 1. We achieve optimal results with the combination: learning rate of  $1 \times 10^{-5}$ , hidden-layer sizes in  $\{200,400\}$  and a batch-size of 16. Refer to Table 7 in Appendix C.
Bi-encoders We fine-tune SBERT with a batch-size of 16, a fixed learning rate of  $2 \times 10^{-5}$ , and AdamW optimizer. Table 8 in Appendix C lists hyper-parameters we initially evaluated.
BM25 and Semantic Search We evaluate for various top  $k$  in  $\{3,\ldots ,18\}$ . We conclude the impact of  $k$  is not big and overall accomplish best results with  $k = 3$  or  $k = 5$  for our experiments. More details in Appendix E.
Evaluation If not otherwise stated, we repeat our in-domain experiments with 10 different random seeds and report mean scores along with standard deviation. For in-domain regression tasks (STS and BWS), we report the Spearman's rank correlation  $(\rho \times 100)$  between predicted and gold similarity scores and for in-domain classification tasks (Quora-QP, MRPC), we determine the optimal threshold from the development set and use it for the test set. We report the  $F_{1}$  score of the positive label. For all domain adaptation tasks, we weakly-label the target domain training dataset and
measure AUC(0.05) as the metric since it is more robust against false negatives (Shah et al., 2018). AUC(0.05) is the area under the curve of the true positive rate as function of the false positive rate  $(fpr)$ , from  $fpr = 0$  to  $fpr = 0.05$ .
Baselines For the in-domain regression tasks, we use Jaccard similarity to measure the word overlap of the two input sentences. For the in-domain classification tasks, we use a majority label baseline. Further, we compare our results against Universal Sentence Encoder (USE) (Yang et al., 2019), which is a popular state-of-the-art sentence embedding model trained on a wide range of training data. We utilise the multilingual model<sup>11</sup>. Fine-tuning code for USE is not available, hence, we utilise USE as a comparison to a large scale, pre-trained sentence embedding method. Further, we compare our data augmentation strategy AugSBERT against a straightforward data augmentation strategy provided by NLPAug, which implements 15 methods for text data augmentation.<sup>12</sup> We include synonym replacement replacing words in sentences with synonyms utilizing a BERT language model. We empirically found synonym-replacement to work best from the rest of the methods provided in NLPAug.
6.1 In-Domain Experiments for AugSBERT
Table 5 summarizes all results for all in-domain datasets. The plain bi-encoder (SBERT w/o Seed
<sup>11</sup>https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3
12 https://github.com/makcedward/nlpaug
Source(Train)	Target(Evaluate)	In-Domain	Cross-Domain
SBERT(Upper-bound)	AugSBERT	SBERT(Lower-bound)	Bi-LSTM(Direct)	Bi-LSTM(Adversarial)
AskUbuntu	Quora	0.504	0.496	0.496	0.059	0.066
Sprint	0.869	0.852	0.747	0.93	0.923
SuperUser	0.802	0.779	0.738	0.806	0.798
Quora	AskUbuntu	0.715	0.602	0.501	0.351	0.328
Sprint	0.869	0.875	0.505	0.875	0.867
SuperUser	0.802	0.645	0.504	0.523	0.485
SuperUser	AskUbuntu	0.715	0.709	0.637	0.629	0.627
Quora	0.504	0.495	0.495	0.058	0.067
Sprint	0.869	0.876	0.785	0.936	0.937
Sprint	AskUbuntu	0.715	0.663	0.613	0.519	0.543
Quora	0.504	0.495	0.496	0.048	0.063
SuperUser	0.802	0.769	0.660	0.658	0.636
Table 6: AUC(0.05) scores for domain adaptation experiments. All except SBERT (in-domain) are evaluated in cross-domain setup with the best transfer strategy highlighted. We adapt (Shah et al., 2018) Bi-LSTM models. Corresponding development set performances can be found in Appendix G, Table 13.
Opt.) consistently underperforms (4.5 - 9.1 points) the cross-encoder across all in-domain tasks. Optimizing the seed helps SBERT more than BERT, however, the performance gap remains open (2.8 - 8.2 points). Training with multiple random seeds and selecting the best performing model on the development set can significantly improve the performance. For the smallest dataset (STS), we observe large performance differences between different random seeds. The best and worst seed for SBERT have a performance difference of more than 21 points. For larger datasets, the dependence on the random seed decreases. We observe bad training runs can often be identified and stopped early using the early stopping algorithm (Dodge et al., 2020). Detailed results with seed optimization can be found in Appendix D.
Our proposed AugSBERT approach improves the performance for all tasks by 1 up to 6 points, significantly outperforming the existing bi-encoder SBERT and reducing the performance difference to the cross-encoder BERT. It outperforms the synonym replacement data augmentation technique (NLPAug) for all tasks. Simple word replacement strategies as shown are not helpful for data augmentation in sentence-pair tasks, even leading to worse performances compared to models without augmentation for BWS and Quora-QP. Compared to the off the shelf USE model, we see a significant improvement with AugSBERT for all tasks except Spanish-STS. This is presumably due to the fact that USE was trained on the SNLI corpus (Bowman et al., 2015), which was used as basis for the
Spanish STS test set, i.e., USE has seen the test sentence pairs during training.
For the novel BWS argument similarity dataset, we observe AugSBERT only gives a minor improvement for cross-topic split. We assume this is due to cross-topic setting being a challenging task, mapping sentences of an unseen topic to a vector space such that similar arguments are close. However, on known topics (in-topic), AugSBERT shows its full capabilities and even outperforms the cross-encoder. We think this is due a better generalization of SBERT bi-encoder compared to the BERT cross-encoder. Sentences from known topics (in-topic) are mapped well within a vector space by a bi-encoder.
Figure 4: Comparison of the density distributions of gold standard with silver standard for various sampling techniques on Spanish-STS (in-domain) dataset.
Pairwise Sampling We observe that the sampling strategy is critical to achieve an improvement using AugSBERT. Random sampling (R.S.) decreases performance compared to training SBERT without any additional silver data in most cases.
BM25 sampling and KDE produces the best AugS-BERT results, followed by Semantic Search (S.S.). Figure 4, which shows the score distribution for the gold and silver dataset for Spanish-STS, visualizes the reason for this. With random sampling, we observe an extremely high number of low similarity pairs. This is expected, as randomly sampling two sentences yields in nearly all cases a dissimilar pair. In contrast, BM25 generates a silver dataset with similar score distribution to the gold training set. It is still skewed towards low similarity pairs, but has the highest percentage of high similarity pairs.  $BM25 + S.S.$ , apart on Spanish-STS, overall performs worse in this combination than the individual methods. It even underperforms random sampling on the BWS and Quora-QP datasets. We believe this is due to the aggregation of a high number of dissimilar pairs from the sampling strategies combined. KDE shows the highest performance in three tasks, but only marginally outperforms BM25 in two of these. Given that BM25 is the most computationally efficient sampling strategy and also creates smaller silver datasets (numbers are given in Appendix F, Table 11), it is likely the best choice for practical applications.
6.2 Domain Adaptation with AugSBERT
We evaluate the suitability of AugSBERT for the task of domain adaptation. We use duplicate question detection data from different (specialized) online communities. Results are shown in Table 6. We can see in almost all combinations that AugSBERT outperforms SBERT trained on out-of-domain data (cross-domain). On the Sprint dataset (target), the improvement can be as large as 37 points. In few cases, AugSBERT even outperforms SBERT trained on gold in-domain target data.
We observe that AugSBERT benefits a lot when the source domain is rather generic (e.g. Quora) and the target domain is rather specific (e.g. Sprint). We assume this is due to Quora forum covering many different topics including both technical and non-technical questions, transferred well by a cross-encoder to label the specific target domain (thus benefiting AugSBERT). Vice-versa, when we go from a specific domain (Sprint) to a generic target domain (Quora), only a slight performance increase is noted.
For comparison, Table 6 also shows the state-of-the-art results from Shah et al. (2018), who applied direct and adversarial domain adaptation with a Bi
LSTM bi-encoder. With the exception of the Sprint dataset (target), we outperform that system with substantial improvement for many combinations.
We presented a simple, yet effective data augmentation approach called AugSBERT to improve bi-encoders for pairwise sentence scoring tasks. The idea is based on using a more powerful cross-encoder to soft-label new sentence pairs and to include these into the training set.
We saw a performance improvement of up to 6 points for in-domain experiments. However, selecting the right sentence pairs for soft-labeling is crucial and the naive approach of randomly selecting pairs fails to achieve a performance gain. We compared several sampling strategies and found that BM25 sampling provides the best trade-off between performance gain and computational complexity.
The presented AugSBERT approach can also be used for domain adaptation, by soft-labeling data on the target domain. In that case, we observe an improvement of up to 37 points compared to an SBERT model purely trained on the source domain.
This work has been supported by the German Federal Ministry of Education and Research (BMBF) under the promotional reference 03VP02540 (ArgumenText), by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1 and grant GU 798/17-1) and has been funded by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE. We would like to thank Andreas Rückle, Jan-Christoph Klie, Mohsen Mesgar, Kevin Stowe and the anonymous reviewers for their feedback.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, et al. 2015. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pages 252-263.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 81-91, Dublin, Ireland. Association for Computational Linguistics.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval-2016. 10th International Workshop on Semantic Evaluation; 2016 Jun 16-17; San Diego, CA. Stroudsburg (PA): ACL; 2016. p. 497-511. ACL (Association for Computational Linguistics).
Giambattista Amati. 2009. BM25, volume 1, pages 257-260. Springer US, Boston, MA.
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2019. Not Enough Data? Deep Learning to the Rescue! arXiv preprint arXiv:1911.03118.
Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory, COLT'98, page 92-100, New York, NY, USA. Association for Computing Machinery.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
John C. Callender and H. G. Osburn. 1979. An empirical comparison of coefficient alpha, guttman's lambda - 2, and msplit maximized split-half reliability estimates. Journal of Educational Measurement, 16(2):89-99.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
Jiecao Chen, Liu Yang, Karthik Raman, Michael Bendersky, Jung-Jung Yeh, Yun Zhou, Marc Najork, Danyang Cai, and Ehsan Emadzadeh. 2020. DiPair: Fast and accurate distillation for trillion-scale text matching and pair modeling. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2925-2937, Online. Association for Computational Linguistics.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670-680, Copenhagen, Denmark. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305.
Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 350-356, Geneva, Switzerland. COLING.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why Does Unsupervised Pre-training Help Deep Learning? Journal of Machine Learning Research, 11:625-660.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations.
Ming Ji, Yizhou Sun, Marina Danilevsky, Jiawei Han, and Jing Gao. 2010. Graph regularized transductive classification on heterogeneous information networks. In Machine Learning and Knowledge Discovery in Databases, pages 570-586, Berlin, Heidelberg. Springer Berlin Heidelberg.
Maurice Kendall. 1948. Rank Correlation Methods. Griffin, Oxford, UK.
David C. Kingsley and Thomas C. Brown. 2010. Preference Uncertainty, Preference Learning, and Paired Comparison Experiments. Land Economics, 86(3).
Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best-worst scaling. In Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 811-817, San Diego, California. Association for Computational Linguistics.
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. *Skip-thought vectors*. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS'15, page 3294-3302, Cambridge, MA, USA. MIT Press.
S. Kullback and R. A. Leibler. 1951. On information and sufficiency. Ann. Math. Statist., 22(1):79-86.
Ashutosh Kumar, Satwik Bhattachamishra, Manik Bhandari, and Partha Talukdar. 2019. Submodular Optimization-based Diverse Paraphrasing and its Effectiveness in Data Augmentation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3609-3619, Minneapolis, Minnesota. Association for Computational Linguistics.
Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. 1998. Efficient BackProp. In Neural Networks: Tricks of the Trade, This Book is an Outgrowth of a 1996 NIPS Workshop, pages 9-50, London, UK, UK. Springer-Verlag.
Amita Misra, Brian Ecker, and Marilyn A. Walker. 2016. Measuring the similarity of sentential arguments in dialogue. In Proceedings of the SIGDIAL 2016 Conference, The 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 13-15 September 2016, Los Angeles, CA, USA, pages 276-287.
Nicole Peinelt, Maria Liakata, and Dong Nguyen. 2019. Aiming beyond the obvious: Identifying non-obvious cases in semantic similarity datasets. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2792-2798, Florence, Italy. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2017. Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338-348, Copenhagen, Denmark. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, and Iryna
Gurevych. 2019. Classification and Clustering of Arguments with Contextualized Word Embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 567-578, Florence, Italy.
Darsh Shah, Tao Lei, Alessandro Moschitti, Salvatore Romeo, and Preslav Nakov. 2018. Adversarial domain adaptation for duplicate question detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1056-1063, Brussels, Belgium. Association for Computational Linguistics.
Christian Stab, Tristan Miller, Benjamin Schiller, Pranav Rai, and Iryna Gurevych. 2018. Cross-topic argument mining from heterogeneous sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, volume Long Papers, pages 3664-3674.
Fabio Henrique Kiyoti dos Santos Tanaka and Claus Aranha. 2019. Data augmentation using GANs. arXiv preprint arXiv:1904.09135.
Antonio Uva, Daniele Bonadiman, and Alessandro Moschitti. 2018. Injecting relational structural representation in neural networks for question similarity. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 285-291, Melbourne, Australia. Association for Computational Linguistics.
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 4144-4150.
Jason Wei and Kai Zou. 2019. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6382-6388, Hong Kong, China. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing.
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional bert contextual augmentation. In Computational Science - ICCS 2019, pages 84-95, Cham. Springer International Publishing.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. 2020. Unsupervised data augmentation for consistency training.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. 2019. Multilingual universal sentence encoder for semantic retrieval. arXiv preprint arXiv:1907.04307.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension. In International Conference on Learning Representations.
In this appendix, we mention the following sections in detail: MTurk guidelines and density distribution analysis for the BWS argument similarity dataset (B), hyperparameter-tuning (C) and seed-optimization (D); provide analysis of the top- $k$  parameter (E) and computational efficiency (F) for our in-domain sampling strategies; report development set performances for all our tasks (G).
B BWS Argument Similarity Dataset
B.1 Amazon Mechanical Turk Guidelines