SlowGuess/NAACL_2021_OCR huggingface dataset:
The annotations required for the BWS Argument Similarity Corpus were acquired via crowdsourcing on the Amazon Mechanical Turk platform. Workers participating in the study had to be located in the US, with more than 100 HITs approved and an overall acceptance rate of  $90\%$  or higher. We paid them at the US federal minimum wage of $7.25/hour. Workers also had to qualify for the study by passing a qualification test consisting of four test questions with argument pairs. Figure 7 exemplifies the instructions given to workers.
B.2 Density Distribution Analysis
Figure 5 compares the density distributions of BWS with Spanish-STS. For the Spanish-STS dataset, the pre-sampling process results in a high amount of pairs towards either ends of the similarity scale—leading to selection bias. The pre-sampling of the creation process of the BWS dataset, in turn, is less biased. There is a much lower number of pairs towards either end of the scale, which is in accordance with data from the wild, i.e. randomly paired arguments.
We implement coarse to fine random search to find the optimal combination of hyperparameters
Figure 5: Comparison of density distribution of BWS Argument Similarity dataset with Spanish-STS dataset.
for both cross-encoders (BERT) and bi-encoders (SBERT). We choose the optimal combination based on the development dataset performance keeping random seed value fixed.[13]
Cross-Encoder (BERT): For all fine-tuning experiments, we optimize a variety of hyperparameters: hidden-layer sizes, learning-rates and batch-sizes. We first evaluate over a wide range of parameters and later conduct a deeper fine search of these optimal parameters. Experimental setup can be found in Table 7.
BERT model	bert-base (uncased/multi.-cased)
hidden layer sizes	{100, 200, 400, 800, 1600, 3200}
Learning rates	{1e-4, 1e-5, 1e-6}
Batch sizes	{8, 16}
Table 7: Experimental setup for hyperparameter tuning of cross-encoder (BERT).
Bi-Encoder (SBERT): For all fine-tuning experiments, we utilize bert-base models, and implement coarse to fine random search with various learning-rates and batch-sizes. Since changing the learning rate scheduler did not contribute to significant improvement, we kept it constant for all our experiments. Experimental setup can be found in Table 8.
D Seed Optimization
For our in-domain tasks, we apply seed optimization i.e. we train our models with 5 random seeds
13Random seed value  $= 42$  during hyperparameter tuning experiments.
BERT model	bert-base (uncased/ multi.-cased)
Learning rates	{2e-5, 1e-6, 1e-7}
Learning rate scheduler	constant
Batch sizes	{8, 16}
Table 8: Experimental setup for hyperparameter tuning of bi-encoder (SBERT).
and select the model that performs best on the development set, and repeat this complete setup 10 times. Testing various seeds can be computationally expensive. In order to reduce the computational overhead, we evaluate whether bad runs can be identified and stopped early. At  $x\%$  of the overall training steps we evaluate the model on the development set and compare the rank with the final ranking of the models on the development set. The results are depicted in Figure 6. We observe a Spearman's rank correlation of over 0.8 at about  $20\%$  of the training steps. We conclude, that bad training runs can often be identified and stopped early.
Figure 6: Spearman's rank correlation for SBERT bi-encoder between development scores at  $x\%$  of the training steps with final development score for in-domain datasets.
E Impact of Top K in Sampling Strategy
In sampling strategies, such as BM25 and semantic search, we are required to pick the top  $k$  values returned by the retrieval engine. Typically for small  $k$  values, positive-pairs are dominant and with increase in  $k$ , negative-pairs start becoming dominant.
We chose a top  $k$  value within  $\{3,5,7,9,12,18\}$  and evaluated the final scores retrieved from our experiments, to measure an impact of  $k$ . Overall, we find the impact of  $k$  to be rather small and  $k = 3$  or  $k = 5$  producing optimal scores for most of the experiments. Top-  $k$  mean test scores for our in-domain datasets are reported in Table 9 for
BM25 and Table 10 for semantic search sampling strategies respectively.
F Computational Efficiency vs. Size of Silver Datasets
The augmented SBERT strategy requires to weakly label a large set of sentence pairs with the cross-encoder. The larger the set of silver pairs, the bigger is the overhead for labeling with the cross-encoder and subsequent training the bi-encoder. Hence, for reasons of efficiency, smaller silver dataset sizes are preferable. Table 11 summarizes the performance of each sampling technique versus the size of sampled silver pairs.
Different sampling strategies create vastly different amounts of sentence pairs. Randomly sampling (R.S.) a large number of sentence pairs is not efficient and often leads to worse performances. KDE with large silver datasets produce optimal scores, but is less computationally efficient. Semantic Search (S.S.) requires the bi-encoder to be additionally trained, which causes computational overhead. Finally, BM25 overall on an average performs best for all tasks given computational efficiency, by sampling out the smallest silver dataset sizes for all tasks in Table 11.
G Development Set Performances
The development set performances for all sentence pair in-domain and domain adaptation tasks can be referred in Table 12 and Table 13 respectively.
Dataset/Top k	Measure	Top 3	Top 5	Top 7	Top 9	Top 12	Top 18
Spanish-STS	ρ × 100	73.67	75.08	74.83	74.71	73.89	72.82
BWS (cross)	ρ × 100	60.02	60.23	61.48	60.65	60.89	61.47
BWS (in)	ρ × 100	66.26	68.63	67.49	67.38	67.74	68.08
Quora-QP	F1	79.01	78.68	78.49	78.40	78.46	77.75
MRPC	F1	85.46	85.17	85.03	84.15	84.24	84.27
Table 9: Summary of In-domain BM25 Sampling Strategy: Top  $k$  mean test scores. We report Spearman's rank correlation  $\rho  \times  {100}$  for regression tasks and  ${F}_{1}$  score for classification tasks.
Dataset/Top k	Measure	Top 3	Top 5	Top 7	Top 9	Top 12	Top 18
Spanish-STS	ρ × 100	73.84	74.22	74.99	74.31	74.22	73.61
BWS (cross)	ρ × 100	60.57	60.51	60.76	61.04	60.74	60.87
BWS (in)	ρ × 100	65.39	68.06	67.01	66.78	66.95	65.93
Quora-QP	F1	77.20	76.65	76.41	76.68	76.33	76.32
MRPC	F1	82.42	82.18	82.20	81.86	81.81	81.91
Table 10: Summary of In-domain Semantic Search Sampling Strategy: Top  $k$  mean test scores. We report Spearman's rank correlation  $\rho  \times  {100}$  for regression tasks and  ${F}_{1}$  score for classification tasks.
Sampling Tech.
Dataset	None	BM25	Sem. Search	BM25 + S.S.	KDE	Random Samp.
Score	(#Silver)	Score	(#Silver)	(Score)	#Silver	(Score)	(#Silver)	(Score)	(#Silver)	(Score)
Spanish-STS	72.07	3,964	75.08	12,715	74.99	16,678	76.24	230,364	74.67	911,072	62.05
BWS (cross-topic)	60.54	11,694	61.48	18,824	61.05	28,771	59.41	559,630	61.49	72,3540	59.95
BWS (in-topic)	63.77	9,236	68.63	11,816	68.06	19,830	63.30	394,252	69.76	565,820	64.54
Quora-QP	74.66	28,014	79.01	47,055	77.20	75,067	72.45	50,147	79.31	1,000,000	73.42
MRPC	84.39	10,637	85.46	18,292	82.39	25,867	82.68	32,353	84.33	1,000,000	82.28
Table 11: Summary of (#silver dataset samples, mean score) for each sampling technique across all in-domain datasets. For STS and BWS datasets, we report the Spearman's rank correlation  $\rho \times 100$  and the  $F_{1}$  score for Quora-QP and MRPC datasets. None represents plain bi-encoder i.e. SBERT. Scores with best sampling strategy and smallest silver dataset size across each dataset are highlighted.
Task
Model / Dataset	(Seed-Opt.)	Regression (ρ × 100)	Classification (F1)
Spanish-STS	BWS (cross-topic)	BWS (in-topic)	Quora-QP	MRPC
Baseline	-	16.98	6.31	5.06	66.67	80.75
BERT	X	89.10 ± 0.69	60.97 ± 1.35	64.89 ± 1.41	81.87 ± 1.07	89.71 ± 0.54
SBERT	X	82.15 ± 0.95	54.77 ± 0.68	62.66 ± 1.06	76.69 ± 0.34	87.30 ± 0.45
BERT (Upper-bound)	✓	88.86 ± 0.74	62.17 ± 0.83	66.23 ± 0.96	81.64 ± 0.99	89.75 ± 0.46
SBERT (Lower-bound)	✓	82.30 ± 1.11	54.90 ± 0.88	62.75 ± 1.16	76.73 ± 0.39	87.37 ± 0.52
SBERT-NLPAug	✓	84.63 ± 0.77	58.66 ± 0.49	66.16 ± 0.41	79.72 ± 0.41	87.24 ± 0.52
AugSBERT-R.S.	✓	75.90 ± 1.89	57.36 ± 0.80	67.83 ± 0.43	73.42 ± 0.74	84.12 ± 0.64
AugSBERT-KDE	✓	85.69 ± 0.54	57.71 ± 0.85	66.14 ± 0.72	80.00 ± 0.31	87.04 ± 0.29
AugSBERT-BM25	✓	85.10 ± 1.11	57.95 ± 0.98	65.37 ± 1.18	77.73 ± 0.47	88.04 ± 0.51
AugSBERT-S.S.	✓	85.28 ± 0.85	56.84 ± 1.22	66.21 ± 0.92	79.29 ± 0.31	83.52 ± 0.27
AugSBERT-BM25+S.S.	✓	85.98 ± 0.75	57.83 ± 0.69	68.45 ± 0.57	79.75 ± 0.24	85.26 ± 0.56
Table 12: Summary of development set scores for the in-domain tasks in this paper. STS and BWS are regression tasks, where we report Spearman's rank correlation  $\rho \times 100$ . Quora-QP and MRPC are classification tasks, where we report  $F_{1}$  score of the positive class. For Baselines, we use a simple Jaccard similarity for regression tasks and a majority label baseline for classification tasks. Scores with the best augmented SBERT strategy are highlighted.
Source(Train)	Target(Evaluate)	In-Domain	Cross-Domain
SBERT(Upper-bound)	AugSBERT	SBERT(Lower-bound)	Bi-LSTM(Direct)	Bi-LSTM(Adversarial)
AskUbuntu	Quora	0.675	0.638	0.496	0.062	0.071
Sprint	0.989	0.773	0.670	0.921	0.917
SuperUser	0.908	0.801	0.586	0.797	0.782
Quora	AskUbuntu	0.844	0.512	0.511	0.328	0.309
Sprint	0.989	0.675	0.524	0.639	0.848
SuperUser	0.908	0.509	0.512	0.529	0.473
SuperUser	AskUbuntu	0.844	0.612	0.564	0.607	0.620
Quora	0.675	0.672	0.496	0.066	0.077
Sprint	0.989	0.848	0.650	0.936	0.933
Sprint	AskUbuntu	0.844	0.724	0.601	0.521	0.532
Quora	0.511	0.668	0.497	0.049	0.063
SuperUser	0.908	0.748	0.620	0.652	0.631
Table 13: AUC(0.05) development scores for domain adaptation experiments. All except SBERT (in-domain) are evaluated in cross-domain setup with the best transfer strategy highlighted. We adapt (Shah et al., 2018) Bi-LSTM models.
Arguments are similar if -
- They say exactly the same thing in different words. Example for topic "Fracking",
Argument A: "And the toxic chemicals associated with fracking operations can contaminate the soil, air and water, and leach into crops".
Argument B: "The chemicals used in fracking are toxic and threaten to poison and pollute our air, ground, water and food supplies - basic necessities for life".
- They cover the same aspect and only differ in minor details. Example for topic "Electric Cars",
Argument A: "With literally hundreds of moving parts, a petro-fired automobile requires considerably more maintenance than an electric car".
Argument B: "Electric cars are much more reliable and require less maintenance than gas-powered cars".
- They talk about the same general aspect but differ in important details. Example for topic "Electric Cars",
Argument A: "Electric cars are environmentally friendly as it reduces air pollution".
Argument B: "Many people think that electric cars are better than gasoline models, not only because of lower operating costs, but because of quicker acceleration and cleaner air".
Arguments are not similar if
- They have the same topic but do not cover the same aspect. Example for topic "Electric cars",
Argument B: "Generally electric motors for automobiles are much easier to maintain".
- They have different topics. Example for topic "Robotic Surgery",
Argument A: "Opponents argue that more drilling offshore could damage sensitive ecosystems".
Argument B: "Robotic surgery offers patients less pain, fewer complications, and a faster return to normal daily activities".
Figure 7: Amazon Mechanical Turk HIT Guidelines used in the annotation study for the BWS Argument Similarity Corpus.