SlowGuess/NAACL_2022_OCR huggingface dataset:
To begin with, we notice that the quality reported on the  $x$ -axis varies only within a relatively narrow range. For instance, the axis ranges from 38.50 through 39.75 for MS MARCO, and all but two of the cheapest settings score above 39.00. Similarly, the  $y$ -axis varies between approximately 50 milliseconds per query up to 250 milliseconds (mostly under 150 milliseconds) using our relatively simple Python-based implementation.
Digging deeper, we see that the best quality in these metrics can be achieved or approached closely with around 100 milliseconds of latency across all three datasets, despite their various sizes and characteristics, and that 2-bit indexing reliably outperforms 1-bit indexing but the loss from more aggressive compression is small.
D LoTTE
Domain coverage Table 9 presents the full distribution of communities in the LoTTE dev dataset.
Figure 4: LoTTE words per passage
Figure 5: LoTTE words per query
The topics covered by LoTTE cover a wide range of linguistic phenomena given the diversity in topics and communities represented. However, since all posts are submitted by anonymous users we do not have demographic information regarding the identify of the contributors. All posts are written in English.
Passages As mentioned in §4, we construct LoTTE collections by selecting passages from the StackExchange archive with positive scores. We remove HTML tags from passages and filter out empty passages. For each passage we record its corresponding query and save the query-to-passage mapping to keep track of the posted answers corresponding to each query.
Search queries We construct the list of LoTTE search queries by drawing from GooAQ queries that appear in the StackExchange post archive. We first shuffle the list of GooAQ queries so that in cases where multiple queries exist for the same answer passage we randomly select the query to include in LoTTE rather than always selecting the first appearing query. We verify that every query has at least one corresponding answer passage.
Forum queries For each LoTTE topic and its constituent communities we first compute the fraction of the total queries attributed to each individual community. We then use this distribution to construct a truncated query set by selecting the
These settings are selected based on preliminary exploration of these parameters, which indicated that performance for larger probe values tends to require scoring a larger number of candidates.
Figure 6: LoTTE answers per query
Corpus	ColBERT	BM25	ANGE	RocketQAY2	SPLADEV2	CobERT2
LoTTE Search Dev Queries (Success@5)
Writing	76.3	47.3	75.7	79.5	78.9	81.7
Recreation	71.8	56.3	66.1	73.0	70.7	76.0
Science	71.7	52.2	66.9	67.7	73.4	74.2
Technology	52.8	35.8	55.7	54.3	56.3	59.3
Lifestyle	73.1	54.4	69.8	72.4	71.2	75.8
Pooled	65.4	45.6	63.7	66.4	67.0	69.3
LoTTE Forum Dev Queries (Success@5)
Writing	75.5	66.2	74.4	75.5	78.1	80.8
Recreation	69.1	56.6	65.9	69.0	68.9	71.8
Science	58.2	51.3	56.3	56.7	59.9	62.6
Technology	39.6	30.7	38.8	39.9	42.1	45.0
Lifestyle	61.1	48.2	61.8	62.0	61.8	65.8
Pooled	59.1	47.8	57.4	58.9	60.6	63.7
highest ranked queries from each community as determined by 1) the query scores and 2) the query view counts. We only use queries which have an accepted answer. We ensure that each community contributes at least 50 queries to the truncated set whenever possible. We set the overall size of the truncated set to be 2000 queries, though note that the total can exceed this due to rounding and/or the minimum per-community query count. We remove all quotation marks and HTML tags.
Statistics Figure 4 plots the number of words per passage in each LoTTE dev corpus. Figures 5 and 6 plot the number of words and number of corresponding answer passages respectively per query, split across search and forum queries.
Dev Results Table 7 presents out-of-domain evaluation results on the LoTTE dev queries. Continuing the trend we observed in 5, ColBERTv2 consistently outperforms all other models we tested.
Licensing and Anonymity The original StackExchange post archive is licensed under a Creative Commons BY-SA 4.0 license (sta). Personal data is removed from the archive before being uploaded, though all posts are public; when we release LoTTE publicly we will include URLs to the original posts for proper attribution as required by the license. The GooAQ dataset is licensed under an Apache license, version 2.0 (Khashabi et al., 2021). We will also release LoTTE with a CC BY-SA 4.0 license. The search queries can be used for non-commercial research purposes only as per the GooAQ license.
E Datasets in BEIR
Table 8 lists the BEIR datasets we used in our evaluation, including their respective license information as well as the numbers of documents as well as the number of test set queries. We refer to Thakur et al. (2021) for a more detailed description of each of the datasets.
Our Touché evaluation uses an updated version of the data in BEIR, which we use for evaluating the models we run (i.e., ColBERTv2 and RocketQAv2) as well as SPLADEv2.
Table 7: Zero-shot evaluation results on the dev sets of the LoTTE benchmark.
Dataset	License	# Passages	# Test Queries
ArguAna (Wachsmuth et al., 2018)	CC BY 4.0	8674	1406
Climate-Fever (Diggelmann et al., 2020)	Not reported	5416593	1535
DBPedia (Auer et al., 2007)	CC BY-SA 3.0	4635922	400
FEVER (Thorne et al., 2018)	CC BY-SA 3.0
FiQA-2018 (Maia et al., 2018)	Not reported	57638	648
HotpotQA (Yang et al., 2018b)	CC BY-SA 4.0	5233329	7405
NFCorpus (Boteva et al., 2016)	Not reported	3633	323
NQ (Kwiatkowski et al., 2019)	CC BY-SA 3.0	2681468	3452
SCIDOCS (Cohan et al., 2020)	GNU General Public License v3.0	25657	1000
SciFact (Wadden et al., 2020)	CC BY-NC 2.0	5183	300
Quora	Not reported	522931	10000
Touché-2020 (Bondarenko et al., 2020)	CC BY 4.0	382545	49
TREC-COVID (Voorhees et al., 2021)	Dataset License Agreement	171332	50
Table 8: BEIR dataset information.
We also tested on the Open-QA benchmarks NQ, TQ, and SQuAD, each of which has approximately 9k dev-set questions and multi-hop HoVer, whose development set has 4k claims. In the compression evaluation §B, we used models trained in-domain on NQ and HoVer, whose training sets contain 79k and 18k queries, respectively.
F Implementation & Hyperparameters
We implement ColBERTv2 using Python 3.7, PyTorch 1.9, and HuggingFace Transformers 4.10 (Wolf et al., 2020), extending the original implementation of ColBERT by Khattab and Zaharia (2020). We use FAISS 1.7 (Johnson et al., 2019) for
$k$ -means clustering, $^9$  though unlike ColBERT we do not use it for nearest-neighbor search. Instead, we implement our candidate generation mechanism (§3.5) using PyTorch primitives in Python.
We conducted our experiments on an internal cluster, typically using up to four 12GB Titan V GPUs for each of the inference tasks (e.g., indexing, computing distillation scores, and retrieval) and four 80GB A100 GPUs for training, though GPUs with smaller RAM can be used via gradient accumulation. Using this infrastructure, computing the distillation scores takes under a day, training a 64-way model on MS MARCO for 400,000 steps takes around five days, and indexing takes approximately two hours. We very roughly estimate an upper bound total of 20 GPU-months for all experimentation, development, and evaluation performed for this work over a period of several months.
Like ColBERT, our encoder is a bert-base-uncased model that is shared between the query and passage encoders and which has 110M parameters. We retain the default vector dimension suggested by Khattab and Zaharia (2020) and used in subsequent work, namely,  $d = 128$ . For the experiments reported in this paper, we train on MS MARCO training set. We use simple defaults with limited manual exploration on the official development set for the learning rate  $(10^{-5})$ , batch size (32 examples), and warm up (for 20,000 steps) with linear decay.
Hyperparameters corresponding to retrieval are explored in §C. We default to probe  $= 2$ , but use probe  $= 4$  on the largest datasets, namely, MS MARCO and Wikipedia. By default we set candidates  $= \text{probe} * 2^{12}$ , but for Wikipedia we set candidates  $= \text{probe} * 2^{13}$  and for MS MARCO we set candidates  $= \text{probe} * 2^{14}$ . We leave extensive tuning of hyperparameters to future work.
We train on MS MARCO using 64-way tuples for distillation, sampling them from the top-500 retrieved passages per query. The training set of MS MARCO contains approximately 800k queries, though only about 500k have associated labels. We apply distillation using all 800k queries, where each training example contains exactly one "positive", defined as a passage labeled as positive or the top-ranked passage by the cross-encoder teacher, irrespective of its label.
We train for 400k steps, initializing from a pre
finetuned checkpoint using 32-way training examples and 150k steps. To generate the top- $k$  passages per training query, we apply two rounds, following Khattab et al. (2021b). We start from a model trained with hard triples (akin to Khattab et al. (2021b)), train with distillation, and then use the distilled model to retrieve for the second round of training. Preliminary experiments indicate that quality has low sensitivity to this initialization and two-round training, suggesting that both of them could be avoided to reduce the cost of training.
Unless otherwise stated, the results shown represent a single run. The latency results in §3 are averages of three runs. To evaluate for Open-QA retrieval, we use evaluation scripts from Khattab et al. (2021b), which checks if the short answer string appears in the (titled) Wikipedia passage. This adapts the DPR (Karpukhin et al., 2020) evaluation code. $^{10}$  We use the preprocessed Wikipedia Dec 2018 dump released by Karpukhin et al. (2020).
For out-of-domain evaluation, we elected to follow Thakur et al. (2021) and set the maximum document length of ColBERT, RocketQAv2, and ColBERTv2 to 300 tokens on BEIR and LoTTE. Formal et al. (2021a) selected maximum sequence length 256 for SPLADEv2 both on MS MARCO and on BEIR for both queries and documents, and we retained this default when testing their system on LoTTE. Unless otherwise stated, we keep the default query maximum sequence length for ColBERTv2 and RocketQAv2, which is 32 tokens. For the ArguAna test in BEIR, as the queries are themselves long documents, we set the maximum query length used by ColBERTv2 and RocketQAv2 to 300. For Climate-FEVER, as the queries are relatively long sentence claims, we set the maximum query length used by ColBERTv2 to 64.
We use the open source BEIR implementation<sup>11</sup> and SPLADEv2 evaluation<sup>12</sup> code as the basis for our evaluations of SPLADEv2 and ANCE as well as for BM25 on LoTTE. We use the Anserini (Yang et al., 2018a) toolkit for BM25 on the Wikipedia Open-QA retrieval tests as in Khattab et al. (2021b). We use the implementation developed by the RocketQAv2 authors for evaluating RocketQAv2.<sup>13</sup>
10https://github.com/facebookresearch/DPR/blob/main/dpr/data/qa_validation.py
11https://github.com/UKPLab/beir
12https://github.com/naver/splade
$^{13}$ https://github.com/PaddlePaddle/RocketQA
<sup>9</sup>https://github.com/facebookresearch/faiss
Topic	Communities	# Passages	# Search queries	# Forum queries
Writing	ell.stackexchange.com	108143	433	1196
literature.stackexchange.com	4778	7	58
writing.stackexchange.com	29330	23	163
linguistics.stackexchange.com	12302	22	116
worldbuilding.stackexchange.com	122519	12	470
Recreation	rpg.stackexchange.com	89066	91	621
boardgames.stackexchange.com	20340	67	179
scifi.stackexchange.com	102561	343	852
photo.stackexchange.com	51058	62	350
Science	chemistry.stackexchange.com	39435	245	267
stats.stackexchange.com	144084	137	949
academia.stackexchange.com	76450	66	302
astronomy.stackexchange.com	14580	15	88
earthscience.stackexchange.com	6734	10	50
engineering.stackexchange.com	12064	16	77
datascience.stackexchange.com	23234	15	156
philosophy.stackexchange.com	27061	34	124
Technology	superuser.com	418266	441	648
electronics.stackexchange.com	205891	118	314
askubuntu.com	296291	132	480
serverfault.com	323943	148	506
webapps.stackexchange.com	31831	77	55
Lifestyle	pets.stackexchange.com	10070	20	87
lifehacks.stackexchange.com	7893	2	50
gardening.stackexchange.com	20601	16	182
parenting.stackexchange.com	18357	10	87
crafts.stackexchange.com	3094	4	50
outdoors.stackexchange.com	13324	16	76
coffee.stackexchange.com	2249	11	50
music.stackexchange.com	47399	65	287
diy.stackexchange.com	82659	135	732
bicycles.stackexchange.com	35567	40	229
mechanics.stackexchange.com	27680	98	246
Table 9: Per-community distribution of LoTTE dev dataset passages and questions.