Experiments	We train and evaluate our models on two passage-ranking datasets, MS MARCO and TREC-CAR.

TREC-CAR	Introduced by  , in this dataset, the input query is the concatenation of a Wikipedia article title with the title of one of its section. The relevant passages are the paragraphs within that section. The corpus consists of all of the English Wikipedia paragraphs, except the abstracts. The released dataset has five predefined folds, and we use the first four as a training set (approximately 3M queries), and the remaining as a validation set (approximately 700k queries). The test set is the same one used to evaluate the submissions to TREC-CAR 2017 (approx. 1,800 queries). Although TREC-CAR 2017 organizers provide manual annotations for the test set, only the top five passages retrieved by the systems submitted to the competition have manual annotations. This means that true relevant passages are not annotated if they rank low. Hence, we evaluate using the automatic annotations, which provide relevance scores for all possible query-passage pairs. We follow the same procedure described for the MS MARCO dataset to fine-tune our models on TREC-CAR. However, there is an important difference.

Introduction	We have seen rapid progress in machine reading compression in recent years with the introduction of large-scale datasets, such as SQuAD  , MS MARCO  , SearchQA  , TriviaQA  , and QUASAR-T  , and the broad adoption of neural models, such as BiDAF  , DrQA  , DocumentQA  , and QAnet  . The information retrieval (IR) community has also experienced a flourishing development of neural ranking models, such as DRMM  , KNRM  , Co-PACRR  , and DUET  . However, until recently, there were only a few large datasets for passage ranking, with the notable exception of the TREC-CAR  . This, at least in part, prevented the neural ranking models from being successful when compared to more classical IR techniques  . We argue that the same two ingredients that made possible much progress on the reading comprehension task are now available for passage ranking task. Namely, the MS MARCO passage ranking dataset, which contains one million queries from real users and their respective relevant passages annotated by humans, and BERT, a powerful general purpose natural language processing model.

Passage Re-ranking with BERT	Recently, neural models pretrained on a language modeling task, such as ELMo (Peters et al., 2017), OpenAI GPT (Radford et al., 2018), and BERT (Devlin et al., 2018), have achieved impressive results on various natural language processing tasks such as question-answering and natural language inference. In this paper, we describe a simple re-implementation of BERT for query-based passage re-ranking. Our system is the state of the art on the TREC-CAR dataset and the top entry in the leaderboard of the MS MARCO passage retrieval task, outperforming the previous state of the art by 27% (relative) in MRR@10. The code to reproduce our results is available at https://github.com/nyu-dl/dl4marco-bert

The official pre-trained BERT models were pre-trained on the full Wikipedia, and therefore they have seen, although in an unsupervised way, Wikipedia documents that are used in the test set of TREC-CAR. Thus, to avoid this leak of test data into training, we pre-trained the BERT re-ranker only on the half of Wikipedia used by TREC-CAR's training set. For the fine-tuning data, we generate our query-passage pairs by retrieving the top ten passages from the entire TREC-CAR corpus using BM25. This means that we end up with 30M example pairs (3M queries * 10 passages/query) to train our model. We train it for 400k iterations, or 12.8M examples (400k iterations * 32 pairs/batch), which corresponds to only 40% of the training set. Similarly to MS MARCO experiments, we did not see any gain on the dev set by training the models longer.

Conclusion	We have described a simple adaptation of BERT as a passage re-ranker that has become the state of the art on two different tasks, which are TREC-CAR and MS MARCO. We have made the code to reproduce our MS MARCO entry publicly available.

Results	We show the main result in Table 1 . Despite training on a fraction of the data available, the proposed BERT-based models surpass the previous state-of-the-art models by a large margin on both of the tasks. We found that the pretrained models used in this work require few training examples from the end task to achieve a good performance 1 . For example, a $\text{BERT}_\text{LARGE}$ trained on 640k question-passage pairs (2% of the MS MARCO training data) is only 2 MRR@10 points lower than model trained on 12.8M pairs (40% of training data).

We use ADAM  with the initial learning rate set to $3 \times 10^{-6}$ , $\beta _1 = 0.9$ , $\beta _2 = 0.999$ , L2 weight decay of 0.01, learning rate warmup over the first 10,000 steps, and linear decay of the learning rate. We use a dropout probability of $0.1$ on all layers.

MS MARCO	The training set contains approximately 400M tuples of a query, relevant and non-relevant passages. The development set contains approximately 6,900 queries, each paired with the top 1,000 passages retrieved with BM25 from the MS MARCO corpus. On average, each query has one relevant passage. However, some have no relevant passage because the corpus was initially constructed by retrieving the top-10 passages from the Bing search engine and then annotated. Hence, some of the relevant passages might not be retrieved by BM25. An evaluation set with approximately 6,800 queries and their top 1,000 retrieved passages without relevance annotations is also provided. We fine-tune the model using TPUs with a batch size of 32 (32 sequences * 512 tokens = 16,384 tokens/batch) for 400k iterations, which takes approximately 70 hours. This corresponds to training on 12.8M (400k * 32) query-passage pairs or less than 2% of the full training set. We could not see any improvement in the dev set when training for another 10 days, which equivalent to seeing 50M pairs in total.

In this paper, we describe in detail how we have re-purposed BERT as a passage re-ranker and achieved state-of-the-art results on the MS MARCO passage re-ranking task.

Question: What is the TREC-CAR dataset?
Answer: in this dataset, the input query is the concatenation of a Wikipedia article title with the title of one of its section. The relevant passages are the paragraphs within that section