Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
Authors: Samuel Humeau
Kurt Shuster
Marie-Anne Lachaux
Jason Weston
Venue:  (2020)

Abstract: The use of deep pre-trained transformers has led to remarkable progress in a number of applications (Devlin et al., 2018). For tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: Cross-encoders performing full self-attention over the pair and Bi-encoders encoding the pair separately. The former often performs better, but is too slow for practical use. In this work, we develop a new transformer architecture, the Poly-encoder, that learns global rather than token level self-attention features. We perform a detailed comparison of all three approaches, including what pre-training and fine-tuning strategies work best. We show our models achieve state-of-the-art results on four tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.

Published as a conference paper at ICLR 2020
POLY-ENCODERS: ARCHITECTURES AND PRE-TRAINING STRATEGIES FOR FAST AND ACCURATE MULTI-SENTENCE SCORING
Samuel Humeau', Kurt Shuster", Marie-Ane Lachaux, Jason Weston
Facebook Al Research
(samuelhumeau kssuster, jase
ABSTRACT
The use of deep pre-trained transformers has led to remarkable progress in num- ber of applications (Devlin et al. 2019). For tasks that make pairwise compar- isons between sequences, matching given input with corresponding label, two approaches are common: Cross-encoder performing full self-attention over the pair and Bi-encoders encoding the pair separately. The former often performs better, but is too slow for practical use. In this work, we develop new trans- former architecture, the Poly-encoder, that learns global rather than token level self-attention features. We perform detailed comparison of all three approaches, including what pre-training and fine-tuning strategies work best. We show our models achieve tfateoof-the-art results four tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.
1 INTRODUCTION
Recently, substantial improvements to state-of-the-art benchmarks on a variety of language under- standing tasks have been achieved through the use of deep pre- preitrained language models followed by fine-tuning (Devlin et al. (2019). In this work we explore improvements to this approach for the class of tasks that require multi-sentence scoring: given an input context, score set of candidate labels, a setup common in retrieval and dialogue tasks, amongst others. Performance in such tasks has to be measured via two axes: prediction quality and prediction speed, as scoring many candidates can be prohibitively slow.
The current ofate-of-the-art focuses on using BERT models for pre-training (Devlin et al. 2019), which employ large text corpora general subjects: Wikipedia and the Toronto Books Corpus (Zhu al. (2055). Two classes of fine-tuned architecture are typically built on top: Bi-encoders and Cross-encoders. Cross-encoders (Wolf et al. 2019 Vig & Ramea 2019), which perform full (cross) self-attention over a given input and label candidate, tend to attain much higher accuracies than their counterparts, Bi-encoders (Mazaré et al. 2018; Dinan et al. 2019), which perform self-attention over the input and candidate label separately and combine them at the end for final representa- separate, Bi-encoders are able to cache the encoded candidates, and
tion. As the representations reuse these representations for each input resulting in fast prediction times. Cross-encoders must recompute the encoding for each input and label; result, they are prohibitively slow test time.
In this work, we provide novel contributions that improve both the quality and speed axes over the current state-of-the-art, We introduce the Poly-encoder, an architecture with an additional learnt at- tention mechanism that represents more global features from which to perform self-attention, result- ing in performance gains over Bi-encoders and large speed gains over Cross Encooder....s. To pre-train our architectures, we show that choosing abundant data more similar to our downstream task also brings significant gains over BERT pre-training. This is true across all different architecture choices and downstream tasks we try.
We conduct experiments comparing the new approaches, in addition to analysis of what works best for various setups of existing methods, on four existing datasets in the domains of dialogue and in- formation retrieval (IR), with pre-training strategies based on Reddit (aazaré et al. 2018 compared
Joint First Authors.

---

Published as a conference paper at ICLR 2020
to Wikipedia/Torome Books (i.e., BERT). We obtain new state-of-the-ar on all four datasets with our best architectures and pre-training strategies, as well as providing practical implementations for real-time use. Our code and models will be released open-source.
2 RELATED WORK
The task of scoring candidate labels given an input context is a classical problem in machine learn- ing. While multi-class classification a special case, the more general task involves candidates as structured objects rather than discrete classes; in this work we consider the inputs and the candidate labels to be sequences of text.
There is a broad class of models that map the input and candidate label separately into a com- mon feature space wherein typically a dot product, cosine or (parameterized) non-linearity is used to measure their similarity. We refer to these models as Bi-encoders. Such methods include vector space models (Salton et al.. 1975). LSI (Deerwester et al.) 1990). supervised embeddings (Bai et al.) 2009 [Wu et 2001) and classical siamese networks (Bromley et al. 1994). For the next utterance prediction tasks we consider in this work, several Bi-encoder neural approaches have been con- sidered, in particular Memory Networks (Zhang al. (0T8a) and Transformer Memory networks (Dinan et al. 2019) as well as LSTMs (Lowe et al.] (0155) and CNNs (Kadlec et al. (2015) which encode input and candidate label separately. A major advantage of Bi-encoder methods is their abil- ity to cache the representations of large, fixed candidate set. Since the candidate encodings are independent of the input, Bi-encoders are very efficient during evaluation.
Researchers have also studied more rich class of models we refer to as Cross-encoders, which make no assumptionns on the similarity scoring function between input and candidate label. Instead, the concatenation of the input and a candidate serve as new input nonlinear function that scores their match based on any dependencies it wants. This has been explored with Sequential Matching Network CNN-based architectures (Wu et al. 2017), Deep Matching Networks (Yang al., 2018 Gated Self- Atte-tttentoon (Zhang et al. 2018b). and most recently transformers (Wolf et al. 2011 Vig & Ramea 2019 Urbanek al. 2019). For the latter, concatenating the sequences of text results in applying self-attention at every layer. This yields rich interactions between the input context and the candidate, as every word in the candidate label can attend to every word in the input context, and vice-versa. [Urbanek et al.] (2019) employed pre-trained BERT models, and fine-tuned both Bi- and Cross-encoders, explicitly comparing them on dialogue and action tasks, and finding that perform better. However, the performance gains come at steep computational cost. Cross-encoder representations are much slower compute, rendering some applications infeasible.
3 TASKS
We consider the tasks of sentence selection in dialogue and article search in IR. The former is task extensively studied and recently featured in two competitions: the Neurips ConvA12 competition (Dinan et al. 2020), and the DSTC7 challenge, Track (Yoshino al., 2019 Jonathan K. Kummer- feld & Lasecki 20188] Chulaka Gunasekara & Lasecki 2019). We compare on those two tasks and addition, we also test on the popular Ubuntu V2 corpus (Lowe al. 2015). For IR, we use the Wikipedia Article Search task of Wu et al. (2018).
The ConvA12 task is based on the Persona-Char dataset (Zhang et al. 20) which involves dia- logues between pairs of speakers. Each speaker is given persona, which is few sentences that describe a character they will imitate, e.g. "I love romantic movies". and is instructed get to know the other. Models should then condition their chosen response on the dialogue history and the lines of persona. As an automatic metric in the competition, for each response, the model has to pick the correct annotated utterance from set of 20 choices, where the remaining 19 were other randomly chosen utterances from the evaluation set. Note that final system however, one would retrieve from the entire training set of over 100k utterances, but this is avoided for speed reasons in common evaluation setups. The best performing competitor out 23 entrants this task achieved 80.7% accuracy on the test set utilizing pre- trained Transformer fine-tuned for this task (Wolf et al. 2019).
The DSTC7 challenge (Track 1) consists of conversations extracted from Ubuntu chat logs, where one partner receives technical support for various Ubuntu-related problems from the other. The
2

---

Published as a conference paper at ICLR 2020
best performing competitor (with 20 entrants in Track 1) in this task achieved 64.5% R@l (Chen & Wang 2019). Ubuntu V2 a similar but larger popular corpus, created before the competition Lowe et al. 2015 report results for this dataset well, as there many existing results on it.
Finally, we evaluate on Wikipedia Article Search (Wu et al. 2018). Using the 2016-12-21 dump of English Wikipedia (~5M articles), the task is given a sentence from an article as search query, find the article it came from. Evaluation ranks the true article (minus the sentence) against 10,000 other articles using retrieval metrics. This mimics web search like scenario where one would like to search for the most relevant articles (web documents). The best reported method is the learning- to-rank embedding model, StarSpace, which outperforms fastText, SVMs, and other baselines.
We summarize all four datasets and their statistics in Table
ConvAl2
DTSC7
Ubuntu V2
Wiki Article Search
Train Ex.
131,4335
100,000
1,000,000
5,035,182
Valid Ex.
7,801
10,000
19,560
9,921
Test Ex.
6634
5,000
18,920
9,925
Eval Cands per Ex.
20
100
10
10,001
Table
Datasets used in this paper.
4
METHODS
In this section we describe the various models and methods that we explored.
4.1 TRANSFORMERS AND PRE-TRAINING STRATEGIES
Transformers Our Bi-, Cross-, and Poly-encoders, described sections 4.2 and 4.4 respec- tively, are based on large pre-trained transformer models with the same architecture and dimension as BERT-base (Devlin et al. 2219),, which has 12 layers, 12 attention heads, and a hidden size of 768. As well as considering the BERT pre-trained weights, we also explore our own pre-training schemes. Specifically, we pre-train two more transformers from scratch using the exact same archi- tecture as BERT-base. One uses a similar training setup as in BERT-base, training on 150 million of examples of [INPUT, LABEL] extracted from Wikipedia and the Toronto Books Corpus, while the other is trained on 174 million examples of [INPUT, LABEL]] extracted from the online platform Reddit (Mazaré et al. 2018). which is a dataset more adapted to dialogue. The former is performed to verify that reproducing a BERT-like setting gives us the same results as reported previously, while the latter tests whether pre-training data more similar to the downstream tasks of interest helps. For training both new setups we used XLM (Lample & Conneau 2019
Input Representation Our pre-training input is the concatenation of input and label [IN- PUT.LABEL], where both are surrounded with the special token [S], following Lample & Conneau 2019). When pre-training on Reddit, the input is the context, and the label is the next utterance. When pre-training on Wikipedia and Toronto Books, as in Devlin et al. (2019), the input one sentence and the label the next sentence in the text. Each input token is represented as the sum of three embeddings: the token embedding, the position (in the sequence) embedding and the segment embedding, Segments for input tokens are O, and for label tokens are 1.
Pre-training Procedure Our pre-training strategy involves training with a masked language model (MLM) task identical to the one in Devlin et al. (2019). In the pre-training on Wikipedia and Toronto Books we add next-sentence prediction task identical to BERT training. the pre-training on Reddit, we add next-utterance prediction task, which is slightly different from the previous one as an utterance can be composed of several sentences. During training 50% of the time the candi- date is the actual next sentence/utterance and 50% of the time it is sentence/utterance randomly taken from the dataset. We alternate between batches of the MLM task and the sext=sentence/next utterance prediction task. Like in Lample & Conneaul (019)) learning rate of 2e-4, Bi 0.9, B2 = 0.98, no L2 weight decay, linear learning rate warmup, and inverse square root decay of the learning rate. We use dropout probability of 0. on all layers, and
the Adam optimizer with
3

---

Published as a conference paper at ICLR 2020
batch of 32000 tokens composed of concatenations [INPUT, LABEL] with similar lengths. We train the model on 32 GPUs for 14 days.
then fine-tune for the multi-sentence selection task of choice, in our case one of the four tasks from Section We consider three architectures with which fine-tune the transformer: the Bi-encoder, Cross-encoder and newly proposed Poly-encoder.
Fine-tuning After pre-training,
4.2 BI-ENCODER
In Bi-encoder, both the input context and the candidate label are encoded into vectors:
Yetur (red(T((ctxi))
Yeand red(T2(cand))
where 7j and T2 are two transformers that have been pre-trained following the procedure described in 4.1 they initially start with the same weights, but are allowed to update separately during fine- tuning. T(x) = ht..., by is the output of a transformer and red(.) function that reduces that sequence of vectors into one vector. As the input and the label are encoded separately, segment tokens are 0 for both. To resemble what is done during our pre-training, both the input and label are surrounded by the special token [S] and therefore h corresponds to [S].
We considered three ways of reducing the output into one representation via red(.): choose the first output of the transformer (corresponding to the special token [S], compute the average over all outputs or the average over the first m s N outputs. We compare them in Table in the Appendix. We use the first output of the transformer our experiments gives slightly better results.
Scoring The score of candidate cand; is given by the dot- dot-product s(ctxt, cand)) = Yetxt" YeandYaand: The network is trained to minimize a cross-entropy loss which the logits are year Yoaud, .... Yorr Yeand, where cand, is the correct label and the others are chosen from the training set. Similar to Mazare jet al. (2018), during training we consider the other labels the batch as negatives. This allows for much faster training, larger batch size; c.g., in our experiments on ConvA12, we were able to use batches 512 elements.
reuse the embeddings computed for each candidate, and also use
Inference speed In the setting of retrieval over known candidates, Bi-encoder allows for the precomputation of the embeddings of all possible candidates of the system. After the context em- bedding Yau is computed, the only operation remaining is dot product between Yeur and every candidate embedding, which can scale to millions of candidates modern GPU, and potentially billions using nearest-neighhor libraries such FAISS (Johnson al. 2019).
4.3
CROSS-ENCODER
The Cross-encoder allows for rich interactions between the input context and candidate label, as they are jointly encoded obtain final representation. Similar to the procedure in pre-training. the context and candidate are surrounded by the special token [S] and concatenated into single vector, which is encoded using one transformer. We consider the first output of the transformer as the context-candidate embedding:
Yexacadd = h1 first(T cand))
where first is the function that takes the first vector of the sequence of vectors produced by the transformer. By using a single transformer, the Cross-encoder is able to perform self-attention be- tween the context and candidate, resulting in a richer extraction mechanism than the Bi-encoder. As the candidate label can attend the input context during the layers of the transformer, the Cross- encoder can produce a candidate-sensitin input representation, which the Bi-encoder cannot. For example, this allows it to select useful input features per candidate.
Scoring To score candidate, a linear layer W is applied to the embedding Yerrzcond to reduce it from a vector to a scalar:
s(ctxt, cand,) Yesr.cond W
Similarly to what done for the Bi-encoder, the network trained to minimize a cross entropy loss where the logits s(ctxt, cand ) s(ctxt, cand,), where cand, is the correct candidate and the
4

---

Published as a conference paper at ICLR 2020
Segre Seduction
Score
E260
Eard
Contont
Candidara
Cut
Out
Out
Out,
Des,
Context Encoder
Candidate Encoder
Encoder
Ir_1
In,
...
1ng
(+) Bi-encoder
(b) Cross-encoder
Legend
Got End
Suore
Aftent
Dang
Learned
Parameter
Emb
Faind avlll
Query
Quary Codemm
Codo
Arlant Noor sgation
Candicate Asgregator
Context Encoder
Candidete Encoder
1n
N,N
1ng
h,
In,
(c) Poly-ancoder
Figure Diagrams of the three model architectures we consider. (a) The Bi-encoder encodes the context and candidate separately, allowing for the caching of candidate representations during inference. (b) The Cross-encoder jointly encodes the context and candidate in single transformer, yielding richer interactions between context and candidate the cost of slower computation. (c) The Poly-encoder combines the strengths of the Bi-encoder and Cross-encoder by both allowing for caching of candidate representations and adding final attention mechanism between global features of the input and given candidate to give richer interactions before computing final score.
others negatives taken from the training set. Unlike in the Bi-encoder, cannot recycle the other labels the batch as negatives, so we use external negatives provided in the training set. The Cross-encodes much more memory than the Bi-encoder, resulting in much smaller batch size.
Inference speed Unfortunately, the Cross-encoder does not allow for precomputation of the can- didate embeddings. At inference time, every candidate must be concatenated with the input context and must go through a forward pass of the entire model. Thus, this method cannot scale to large amount of candidates. We discuss this bottleneck further in Section 5.4
4.4 POLYEENCODER
The Poly-encoder architecture aims to get the best of both worlds from the Bi- and Cross-encoder. given candidate label is represented by one vector as the Bi-encoder, which allows for caching candidates for fast inference time, while the input context is jointly encoded with the candidate, as the Cross-encoder, allowing extraction of more information.
The Poly-encoder uses two separate transformers for the context and label like a Bi-encoder, and the candidate is encoded into a single vector Yeond, As such, the Poly-encoder method can be im- plemented using precomputed cache of encoded responses. However, the input context, which is typically much longer than a candidate, is represented with m vectors (y- (y instead of just one in the Bi-encoder, where will influence the inference speed. To obtain these global features that represent the input, we learn m context codes (CI. Cm), where e extracts representation y by attending over all the outputs of the previous layer. That is, we obtain y' using:
Yau Ewghy
where (Wf..., w = softmax(c; hi.., C; hx)

---

Published as a conference paper at ICLR 2020
The context codes randomly initialized, and learnt during finetuning. Finally, given our m global context features, we attend over them using Yeond, as the query:
Yeert - WaYEaE
where (Wi,-., Wm) softmax(ycomd, Yeerr in Yeand; Yews
The final score for that candidate label is then Yeur Yeand, as in Bi-encoder. As m N, where N is the number of tokens, and the context-candidate attention is only performed at the top layer, this is far faster than the Cross-encoder's full self-attention.
5 EXPERIMENTS
We perform variety of experiments to test our model architectures and training strategies over four tasks. For metrics, we measure Recall@k where each test example has C possible candidates to select from, abbreviated to @@k as well as mean reciprocal rank (MRR).
5.1 BI-ENCODERS CROSS-ENCODERS
We first investigate fine-tuning the Bi- and Cross-encoder architectures initialized with the weights provided by Devlin et al. ),019), studying the choice of other hyperparameters (we explore our own pre-traim pre-training schemes in section 3.3). In the case of the Bi-encoder, we can use large number of neg- atives by considering the other batch elements as negative training samples, avoiding recomputation of their embeddings. On 8 Nvidia Volta v100 GPUs and using half-precision operations (i.e. float operations), we can reach batches of 512 elements on ConvA12. Table P shows that in this setting, obtain higher performance with larger batch size, i.c. more negatives, where 511 negatives yields the best results. For the other tasks, we keep the batch size at 256, as the longer sequences memory. The Cross-encoder is more computationally intensive, as the
in those datasets embeddings for the (context, candidate) pair must be recomputed each time. We thus limit its batch size 16 and provide negatives random samples from the training set. For DSTC7 and Ubuntu V2, we choose 15 such negatives; For ConvA12, the dataset provides 19 negatives.
Negatives
31
63
127
255
511
R@ 1/20
81.0
81.7
82.3
83.0
83.3
Table 2: Validation performance on ConvA12 after fine-tuning a Bi-encoder pre-trained with BERT, averaged over 5 runs. The batch size is the number of training negatives + I as we use the other elements of the batch as negatives during training.
The above results are reported with Bi-encoder aggregation based on the first output. Choosing the average over all outputs instead is very similar but slightly worse (83.1, averaged over 5 runs). We also tried to add further non-linearities instead of the inner product of the two representations, but could not obtain improved results over the simpler architecture (results not shown).
We tried two optimizers: Adam (Kingma & Ba, 2015) with weight decay of 0.01 (as recommended by (Devlin al. 2019)) and Adamax (Kingma Ba 2015) without weight decay; based on val- idation set performance, we choose to fine-tune with Adam when using the BERT weights. The learning rate is initialized to 5e-5 with warmup of 100 iterations for Bi- and Poly-encoders, and 1000 iterations for the Cross-encoder. The learning rate decays by factor of 0.4 upon plateau of the loss evaluated on the valid set every half epoch. In Table B we show validation performance when fine-tuning various layers of the weights provided by (Devlin et al. 2019)) using Adam with decay optimizer. Fine-tuning the entire network is important, with the exception of the word embeddings.
With the setups described above, we fine-tune the Bi- and Cross-encoders on the datasets, and report the results Table 4 On the first three tasks, our Bi-encoders and Cross-encoders outperform the best existing approaches in the literature when we fine-tune from BERT weights. E.g., the Bi- encoder reaches 81.7% R@1 I on ConvA12 and 66.8% @ on DSTC7, while the Cross-encoder achieves higher scores of 84.8% R@1 I on ConvAl2 and 67.4% R@1 I on DSTC7. Overall, Cross- encoders outperform all previous approaches on the three dialogue tasks, including our Bi-encoders (as expected). We do not report fine-tuning of BERT for Wikipedia IR cannot guarantee the

---

Published as a conference paper at ICLR 2020
Fine-tuned parameters
Bi-encoder
Cross-encoder
Top layer
74.2
80.6
Top layers
82.0
86.3
All but Embeddings Every Layer
83.3
87.3
83.0
86.6
Table 3:
Validation performance (R@ @ 1/20) on ConvA12 using pre-trained weights BERT-base with different parameters fine-tuned. Average over 5 runs (Bi-encoders) or runs (Cross-encoders).
test is not part of the pre-training for that dataset. In addition, Cross-encoders are also too slow to evaluate on the evaluation setup of that task, which has 10k candidates.
Dataset
ConvA12 test
DSTC
Ubuntu v2
Wikipedia IR
split
Test
test
test
metric
R@ 1/20
R@ 1/T00
MRR
R
MRR
R@ R@1/10001
Wolf et al.
2019)
80.7
Gu et al. 2018)
60.8
69.1
Chen & Wang
(019)
64.5
73.5
Yoon al. 2018)
65.2
Dong & Huang 2018)
75.9
84.8
Wu et al. 2018
56.8
pre-trained BERT weights from
Devlin
2019)
Toronto Books
Wikipedia
Bi-encoder
81.7 = 0.2
66.8 = 0.7
74.6 0.5 80.6 = 0.4
88.0 + 0.3
Poly-encoder 16
83.2 = 0.1
67.8 + 0.3
75.1 3.1+0.2. 81.2 * 0.2
88.3 + 0.1
Poly-encoder 64
83.7 0.2
67.0 + 0.9
81.3 0.6
0.2
88.4 0.1
Poly-encoder 360
83.7 = 0.2
68.9 # 0.4
76,2 1.2
80.9 = 0.0
88.1 = 0.1
Cross-encoder
84.8 # 0.3
67.4 0.7
75.6 + 0.4
82.8 + 0.3
89.4 + 0.2
Our pre pre-training Toronto Books
Wiikipedia
Bi-encoder
82.0 0.1
64.5 + 0.5
72.6 + 0.4
80.8 = 0.5
88.2 0.4
Poly-encoder 16
82.7 a 0.1
65.3 0.9
73.2 + 0.7
83.4 # 0.2
89.9 0.1
Poly-encoder 64
83.3 0.1
6 0.7
73.5 0.5
83.4 0.1
89.9 0.0
Poly-encoder 360
83.8
65.8 0.7
0.6
83.7 =
90.1 =
Cross-encoder
84,9 # 0.3
65.3 a 1.0
73.800.6
83.1 # 0.7
89.7 + 0.5
Our pre-training Reddit Bi-encoder
84.8 = 0.1
70.9 + 0.5
78.1
83.6 0.7
90.1 0.4
71.0
Poly-encoder 16
86.3 = 0.3
71.6 0.6
78.4
86.0 + 0.1
91.5
71.5
Poly-encoder 64
86.5 0.2
0.8
78.2 0.7
85.9 * 0.1
91.5 0.1
71.3
Poly-encoder 360
86.8 0.1
71.4 1.0
78.3 + 0.7
85,9 =
91.5 =
71.8
Cross-encoder
87.9 = 0.2
71.7 * 0.3
79.0 + 0.2
86.5 + 0.1
91.9 + 0.0
Table 4: Test performance of Bi-, Poly- and Cross-encoders on our selected tasks.
5.2 POLY-ENCODERS
We train the Poly-encoder using the same batch sizes and optimizer choices as in the Bi-encoder experiments. Results are reported Table various values of context vectors.
The Poly-encoder outperforms the Bi-encoder on all the tasks, with more codes generally yielding larger improvements. Our recommendation thus to use as large a code size as compute time allows (see Sec. [.4). On DSTC7, the Poly-encoder architecture with BERT pretraining reaches 68.9% RI with 360 intermediate context codes; this actually outperforms the Cross-encoder result (67.4%) and is noticeably better than our Bi-encoder result (66.8%). Similar conclusions are found Ubuntu V2 and ConvA12, although in the latter Cross-encoders give slightly better results.
We note that since reporting results, the authors of Li al. 2019)) have conducted human evaluation study on ConvA12, which our Poly-encoder architecture outperformed all other models compared against, both generative and retrieval based, including the winners of the competition.
7

---

Published as a conference paper at ICLR 2020
Scoring time (ms)
CPU
GPU
Candidates
1k
100k
1k
100k
Bi-encoder
115
160
19
22
Poly-encoder 16
122
678
18
38
Poly-encee Poly-encoder 64
126
692
23
46
Poly-encoder 360
160
837
57
88
Cross-encoder
21.7k
2.2M
2.6k
266k
Table 5: Average time in milliseconds to predict the next dialogue utterance from C possible candi- dates on ConvA12, * are inferred.
5.3 DPMAIN-SPECITIC PRE-TRAINING
We fine-tune our Reddit-pu teiiit-ppeeetteeeee pre-trained transformer on all four tasks; we additionally fine-tune trans- former that was pre-trained on the same datasets as BERT, specifically Toronto Books Wikipedia. When using our pre-trained weights, we use the Adamax optimizer and optimize all the layers of the transformer including the embeddings. As we not use weight decay, the weights of the final layer much larger than those in the final layer of BERT; to avoid saturation of the attention layer in the Poly-encoder, we re-scaled the last linear layer so that the standard deviation of its output matched that of BERT, which we found necessary to achieve good results. We report results of tine-tuning with our pre-trained weights in Table 4 the-art performance over our previous results with BERT, finding that see for all three dialogue tasks, and all three architectures.
We show that pre-training on Reddit gives further state-of-
The results obtained with fine-tuning on our own transformers pre-trained Toronto Books Wikipedia are very similar to those obtained with the original BERT weights, indicating that the choice of dataset used to pre-train the models impacts the final results, not some other detail in our training. Indeed, as the two settings pre-train with datasets of similar size, we can conclude that choosing a pre-training task (e.g. dialogue data) that is similar to the downstream tasks of interest (e.g. dialogue) is likely explanation for these performance gains, in line with previous results showing multi-tasking with similar tasks more useful than with dissimilar ones (Caruana 1997).
5.4 INFERENCE SPEED
An important motivation for the Poly-encoder architecture is to achieve better results than the Bi- encoder while also performing at a reasonable speed. Though the Cross-encoder generally yields strong results, it is prohibitively slow. We perform speed experiments to determine the trade-off of improved performance from the Poly-encoder. Specifically, we predict the next utterance for 100 dialogue examples in the ConvA12 validation set, where the model scores C candidates (in this case, chosen from the training set). We perform these experiments on both CPU-only and GPU setups. CPU computations were run on an 80 core Intel Xeon processor CPU E5-2698. GPU computations were run on single Nvidia Quadro GP100 using cuda 10.0 and cudnn 7.4.
We show the average time per example for each architecture in Table 5 The difference in timing between the Bi-encoder and the Poly-encoder architectures is rather minimal when there are only 1000 candidates for the model to consider. The difference more pronounced when considering 100k candidates, a more realistic setup, Nevertheless, both models are still tractable. The Cross-encoder, however, is 2 orders of magnitude slower than the Bi-encoder and Poly-encoder, rendering it intractable for real-time inference, e.g- when interacting with a dialogue agent, or retrieving from a large set of documents. Thus, Poly- encoders, given their desirable performance and speed trade-off, are the preferred method.
5-6x slowdown for the Poly-encoder variants.
We additionally report training times in the Appendix, Table 6 Poly-encoders also have the benefit of being 3-4x faster to train than Cross-encoders (and are similar in training time Bi-encoders).
8

---

Published as a conference paper at ICLR 2020
6 CONCLUSION
In this paper we present new architectures and pre-training strategies for deep bidirectional trans- formers in candidate selection tasks. We introduced the Poly-encoder method, which provides mechanism for attending over the context using the label candidate, while maintaining the ability to precompute each candidate's ' representation, which allows for fast real-time inference in a produc- tion setup, giving improved trade off between accuracy and speed. We provided experimental analysis of those trade-offs for Bi-, Poly- and Cross-encoders, showing that Poly-encoders are more accurate than Bi-encoders, while being far faster than Cross-encoders, which are impractical for real-time use. In terms of training these architectures, we showed that pre-training strategies more closely related the downstream task bring strong improvements. In particular, pre-training from scratch on Reddit allows us to outperform the results we obtain with BERT, a result that holds for all three model architectures and all three dialogue datasets we tried. However, the methods introduced in this work are not specific to dialogue, and can be used for any task where one is scoring set of candidates, which we showed for information retrieval task as well.
REFERENCES
Bing Bai, Jason Weston, David Grangier, Ronan Collobert, Kunihiko Sadamasa, Yanjun Qi, Olivier Chapelle, and Kilian Weinberger. Supervised semantic indexing. In Proceedings of the 18th ACM conference on Information and knowledge management, pp. 187-196. ACM, 2009.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verifi- cation using a" siamese" time delay neural network. In Advances in neural information processing systems, pp. 737-744, 1994.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Qian Chen and Wen Wang. Sequential attention-based network for noetic end-to-end response selection. CoRR, abs/1901 abs/1901.02609, 2019. U.LJitpp///axxvvorrr///////.... 02609
Lazaros Polymenakos Chulaka Gunasekara, Jonathan K. Kummerfeld and Walter S. Lasecki. Dstc7 task I: Noetic end-to-end response selection. In 7th Edition of the Dialog System Technol- ogy Challenges at AAAI 2019, January 2019. URL http://wookkhhopcolll...o.og final report. fiwall_eport..ppppp.
popeer/dstcc_eeee
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, (6):391-407 1990.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: (0.18653/v1/N199 1423.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of Wikipedia: snowledge-poweree conversational agents. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. The second conversational intelligence challenge (convai2). In Sergio Escalera and Ralf Herbrich (eds.). The NeurIPS '18 Competition, pp. 187-208, Cham, 2020. Springer International Publish- ing. ISBN 978-3-030-29135.2
Jianxiong Dong and Jim Huang. Enhance word representation for out on ubuntu dialogue corpus. CoRR, abs/1 011188022221144 2018. URL http:://aww...rg/aa/b/////////// 02614

---

Published as a conference paper at ICLR 2020
Jia-Chen Gu, Zhen-Hua Ling, Yu-Ping Ruan, and Quan Liu. Building sequential inference models for end-to-end response selection. CoRR, abs/1812.00666, 2018. URL http:://rxiv..rrt abs/ 1812. 00686
J. Johnson, M. Douze, and H. Jgou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, pp. 1-1, 2019. ISSN 2372-2096. doi: 10. 10.1109/TBAATA.200 201..2221211222255222
Joseph Peper Vignesh Athreya Chulaka Gunasekara Jatin Ganhotra Siva Sankalp Patel Lazaros Poly- menakos Jonathan K. Kummerfeld, Sai R. Gouravajhala and Walter S. Lasecki. Analyzing as- sumptions in conversation disentanglement research through the lens of a new dataset and model. ArXiv e-prints, October 2018. URL
wwtp:oogggppp.../oe 11118.11111111111111111111111111 111...pff
Rudolf Kadlec, Martin Schmid, and Jan Kleindienst. Improved deep learning baselines for ubuntu corpus dialogs. CoRR,
2015.
/111 03753
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICIR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
6980
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Margaret Li, Jason Weston, and Stephen Roller. Acute-eval: Improved dialogue evaluation with optimized questions and multi-tune comparisons. arXiv preprint (atp://990.00777 2019.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-tur turn dialogue systems. In SIGDIAL Conference, 2015.
Pierre-Emmannu Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. Training millions of personalized dialogue agents. In EMNLP, 2018
Gerard Salton, Anita Wong, and Chung-Shu Yang. A vector space model for automatic indexing. Communications of the ACM, 18(11):63-662, 1975.
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. Learning to speak and act in fantasy text adventure game. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)), pp. 673-683, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. (0.18653/v/DDI9-1062
Jesse Vig and Kalai Ramea. Comparison of transfer-leaming approaches for response selection in multi-turn conversations. Workshop on DSTC7, 2019.
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint UrXiv: 1494 2019.
Ledell Yu Wu, Adam Fisch, Sumit Chopra, Keith Adams, Antoine Bordes, and Jason Weston. Starspace: Embed all the things! In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Yu Ping Wu, Wei Chung Wu, Chen Xing, Ming Zhou, and Zhoujun Li. Sequential matching net- work: A new architecture for multi-turn response selection in retrieval-based chatbots. In ACL, 2017.
Liu Yang, Minghui Qiu, Chen Qu, Jiafeng Guo, Yongfeng Zhang, Bruce Croft, Jun Huang, and Haiqing Chen. Response ranking with deep matching networks and external knowledge in information secking conversation systems. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 245- 254. ACM, 2018.
10

---

Published as a conference paper at ICLR 2020
Seunghyun Yoon, Joongbo Shin, and Kyomin Jung. Learning rank question- -nsswer pairs using hierarchical recurrent encoder with latent topic clustering. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long Papers), pp. 1575-1584, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18655/11/N1--11
Koichiro Yoshino, Chiori Hori, Julien Perez, Luis Fernando Haro, Lazaros Polymenakos, R. Chu- laka Gunasekara, Walter S. Lasecki, Jonathan K. Kummerfeld, Michel Galley, Chris Brockett, Jianfeng Gao, William B. Dolan, Xiang Gao, Huda AlAmri, Tim K. Marks, Devi Parikh, and Dhruv Batra. Dialog system technology challenge 7. CoRR, abs/1901.03445 2019.
Saizheng Zhang, Emily Dinan, Jack Urbanck, Arthur Szlam, Douwe Kiela, and Jason Weston. Per- sonalizing dialogue agents: have dog. do you have pets too? In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics, pp. 2204-2213, Melhourne, Australia, July 201 Association for Computational Linguistics.
Zhuosheng Zhang, Jiangtong Li. Pengfei Zhu, Hai Zhao, and Gongshen Liu. Modeling multi-turn conversation with deep utterance aggregation. In COLING, 2018b.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan R. Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 19-27, 2015.

---

Published as a conference paper at ICLR 2020
A
TRAINING TIME
We report the training time on GPU Volta 100 for the datasets considered and for 4 types of models in T
Dataset
ConvA12
DSTC7
Vb
Bi-encoder
2.0
4.9
7.9
Poly-encoder 16
2.7
5.5
8.0
Poly-encoder 64
2.8
5.7
8.0
Cross-enceder64
9.4
13.5
39.9
Table 6: Training time in hours.
B REDUCTION LAYER IN BI-ENCODER
We provide in Table the results obtained for different types of reductions on top of the Bi-encoder. Specifically compare the Recall @ 1/20 the Conv A12 validation set when taking the first output of BERT, the average of the first 16 outputs, the average of the first 64 outputs and all of them except the first one ([S)).
Setup
ConvAl2 valid Recall 1/20
First output
Avg first 16 outputs
82.9
Avg first 64 outputs
82.7
Avg all outputs
83.1
Table 7: Bi-encoder results the ConvA12 valid set for different choices of function red(.).
C ALTERNATIVE CHOICES FOR CONTEXT VECTORS
We considered a few other ways to derive the context vectors Ouum..,, Yan) of the Poly-encoder from the output (h) (e. ( of the underlying transformer:
Learn m codes (CI,..., Cmn, where extracts representation You by attending over all the outputs (h .... ) This method denoted "Poly-encoder (Learnt-cdess"" or "Poly- encoder (Learnt-m)" and is the method described in section 4.4
Consider the first m outputs (h .... Mm This method is denoted "Poly-encoder (First m outputs)" or "Poly-encoder (First-m)". Note that when N m, only vectors are considered.
Consider the last m outputs.
Consider the last m outputs concatenated with the first one, h which plays a particular role in BERT it corresponds to the special token [S].
The performance of those four methods is evaluated on the validation set of Convai2 and DSTC7 and reported on Table The first two methods are shown Figure We additionally provide the inference time for a given number of candidates coming from the Convai2 dataset on Table 9
12

---

Published as a conference paper at ICLR 2020
Dataset
ConvA12
DSTC
split
dev
test
dev
test
metric
@@ 1/20
R @ 1/20
@@ 1/100
R@ 1/100
(Wolf et al. 2019)
82.1
80.7
(Chen & Wang 2019)
57.3
64.5
Attention Code
Learnt-codes
81.9 = 0.3
81.0 +.10
56.2 0.1
66.9 0.7
First outputs
83.2 = 0.2
81.5
56.4
66.8 + 0.7 67.2 + 1.1
Last m outputs
82.9 = 0.1
81.0 + 0.1
56.1 + 0.4
Last m outputs and b
4 Attention Codes
Learnt-codes
8388 = 0.2
82.2 + 0.5
56.3 0.5
66.8 + 0.7
First m outputs
83.4 0.2
0.1
56.9 0.5
67.2 1.3
Last m outputs
82.8 = 0.2
81.3 0.4
56.0
65.8 + 0.5
Last m outputs and h!
82.9 =
81.4 0.2
55.8 0.3
66.1 + 0.8
16 Attention Codes
Learnt-codes
84.4
0.1
57.7 = 0.2
67.8 = 0.3
First m outputs
85.2 0.1
83.9 0.2
56.1 1.7
66.8 1.1
Last m outputs
83.9 = 0.2
82.0 22.000.4 0.4
56.1 0.3
66.2 + 0.7
Last m outputs and h
83.8 = 0.3
81.7 + 0.3
56.1 + 0.3
66.6 + 0.2
64 Attention Codes
Learnt-codes
84.9 0.1
83.7 0.2
58.3 0.4
67.0 0.9
First m outputs
86.0 = 0.2
84.2 0.2
57.7 $7.7
67.1 0.1
Last outputs
84.9 = 0.3
82.9
57.0 0.2
66.5 0.5
Last m outputs and h
85.0 = 0.2
83.2 0.2
57.3 # 0.3
67.1 + 0.5
360 Attention Codes
Learnt-codes
85.3 0.3
57.7 8.2
68.9 0.4
First m outputs
86.3 = 0.1
0.3
58.1 0.4
66.8 + 0.7
Last outputs
86.3 = 0.1
84.7 0.3
58.0 0.4
68.1 0.5
Last outputs and h
86.2 = 0.3
84.5 + 0.4
58.3 0.4
68.0 0.8
Table 8: Validation and test performance of Poly-encoder variants, with weights initialized from
(Devlin et al.) 2019). Scores are shown for ConvA12 and DSTC Track Bold numbers indicate the highest performing variant within that number of codes.
Scoring time (ms)
CPU
GPU
Candidates
lk
100k
1k
100k
Bi-encoder
115
160
19
22
Poly-encoder (First outputs) 16
119
551
17
37
Poly encoder (First mn outputs) 64
124
570
17
39
Poly-encoder (First m outputs)
360
120
619
17
45
Poly-encoder (leeartt-codes) 16
122
678
18
38
Poly-encodes (Learnt-codes) 64
126
692
23
46
Poly-encoder (l.carnt-codes) 360
160
837
57
88
Cross-encoder
21.7k
2.2M
2.6k
266k
Table 9: Average time in milliscconds to predict the next dialogue utterance from N possible candi- dates. are inferred.
13

---

Published as a conference paper at ICLR 2020
Legend
Token
Vector
Allertiooy
Poromettr
Approgation
Score katuction
Score
SandErm
Sard Erad
Contort: Aggregator
Candidanu Aug
Aggrngeee
D...
K
Out
Context Encoder
Candidate Encoder
Encoder
1n,2
Bi-encoder
(b) Cross-encoder
Coe End)
Score
Do Fri
Absnnt
Aenn on.
Duery
Query
Bmib
Erit
anddiam
Oe.
Query
Code
After tiaon
Artentian
Aggregator
Aggregator
Get.
Out
Out
Det.
N.
Ovt,
Context Encoder
Condidate Encoder
Context Encoder
Candidate Encoder
n,1
11
In,1
Poly-encoder (First-m)
Poly-encoder (Learnt-m)
Figure 2: (a) The Bi-encoder (b) The Cross-encoder (c) The Poly-encoder with first m vectors. (d) The Poly-encoder with m learnt codes.
Dotaset
ConvA12
DSTC7
Uhuntu v2
split
dev
sest
dev
test TQTOOI
dev 1/10
lest Re510
ieemic
1,20
1/100
1/100
R/11/10
Hugning Wall al 2019)
82.1
80.7
Chin Wane, 20191
90.2
[ong & Hlunng 2018
759
973
$1.3
[Ne-tiined weights from
Devlin at
2019
Toronto
Books
Wikipedia
Bi-enookr
83.3
0.2
563
0.44
0.7
89.0 1.0
809
109600
30.6
98.2 2
Poly-cnooder (First-m)
8.12+0.1
56.7
88.
98.2
88.5
Poly-emooder (Leumi-m) l6
0..4
332201
31.7
2.2
67.8
0.3
0.2
15.1
81.3
81.2
982 2
B5.3
Poly-znodder
04
98.4
Poly-eneoder [Leumt-m] 64
849 0.1
33.7 0.2
38.3 0.4
61.0 09
89.2 0.2
74.7 0.6
81.3 001
98.2
58.4
Poly-meexer (First-m) 360
0.1
0.3
0.5
6
09
750
82.7
82.2
0.6
98.4
89.0 0.4
Poly-cnonder (Lcumt-m)
0.5
37.7
Cross.encods encoder
87.1
$43 0.3
0.44
67.4 0.7
90.5
1...0...
833 0.4
$2.8 2 0.3
98.4 0.1
89.4 0.2
Our noe-traiing on Toronto Books Wilipedia
Bi-encoder
Paly-eneader (First-m)
84.1=02
514+002
33.9
63.3 29
81.2 13
716 2.4
50X 0.0
90.6
98.4
S8.1 0.3
Fuly eneoder [Leumt l6
0.2
32. 2.1
2..0
03
0.72
03
840:0.
83.4
98.7
19.9
Poly-onooder (First-m)
86.
87.8
Paly- [Leamt ml 64
0.1
33. 0.1
36.2 0.4
6.78
88.4 0.3
75.5 0.5
84.
83.4 0.1
98.7 =
89.9 0.0
encoder
57.5
81.
Paly-encoder [Leumt-m] 360
86.1
0.1
30.3
a 0.
65.8 =
88.5
7..6
842.02
0.2
83.7
98.7 =
90.1 0.0
Cross
$19.00
37.7 0.3
63.3
0..
73.6200.6
83. 03
83.12
98.7 0.1
197.7
Our pnc-traimin on Redun
Bi-encader
8699
8.1
60.1 0.4
70.9 0.5
0..
83.7 0.3
83.6 0.7
98.8
90.1 0.4
Poly-enooder (First-m)
91.0
Poly-enewder (Leumi-m) 16
8866
61.1 0.4
71.6
91.3 0.3
7.4
86.1
0.1
86.0 0.1
90.0 0.1
91.5 0.1
Poly-zneoder (First-m)
895
073
01.3=00
0.0
840
83.9
0.3
Poly-eneoder (Leumt-nn 64
89.0
$65
009 0.6
71.2
91.3
0.7
0.1
8..9
99.1 0.0
31.5 0.1
Poly-cnewder (First-m) 360
90.0 0.1
07
61.1 1.9
7099 21
91.5
77.9 1.6
848 0.5
34.6 0.5
98.9 0.1
90.7 + 0.3
Poly-cnoodes (Leumt-m)
Ches-encedeer
90.3 0.2
07
63.9
71.7
03
92
70
867+0..
86.5 0.1
90.1
0.0
91.9 0.0
Table 10: Validation and test performances of Bi-, Poly- and Cross-encoders. Scores are shown for
ConvA12, DSTC7 Track and Ubuntu v2, and the previous state-of-the-ar models in the literature.
14


Reviews:
[{"review_id": "HJeESBliqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes an approach for scoring sentences based on pre-trained transformers. The paper describes two main existing approaches for this task, namely bi-encoders and cross-encoders and then proposes a new formulation called poly-encoders which aims to sit between the existing approaches offering high accuracy -- similarly to cross-encoders -- and high efficiency -- similarly to bi-encoders. The paper is well written and although this is not related to my research I enjoying reading it. The approach proposed seems reasonable to me, and of sufficient novelty while the results presented are impressive. Moreover the paper seems a good fit for ICLR.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1xLuTzbcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new neural network architecture based on transformers called poly-encoders. These are compared against many state-of-the-art approaches including cross-encoders and bi-encoders on many large-scale datasets. \nBi-encoders > Poly-encoders > Cross-encoders in terms of speed and \nCross-encoders > Poly-encoders > Bi-encoders in terms of accuracy.\n\nI am not an expert in this area. However, to the best of my knowledge I don't see anything immediately wrong with this. The experiments are also comprehensive. Therefore I recommend acceptance.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByeUgZT35r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This work proposes a new transformer architecture for tasks that involve a query sequence and multiple candidate sequences. The proposed architecture, called poly-encoder, strikes a balance between a dual encoder which independently encodes the query and candidate and combines representations at the top, and a more expressive architecture which does full joint attention over the concatenated query and candidate sequences. Experiments on utterance retrieval tasks for dialog and an information retrieval task show that poly-encoders strike a good trade-off between the inference speed of the dual encoder model and the performance of the full attention model. \n\nPros:\n- Strong results compared to baselines on multiple dialog and retrieval tasks. \n- Detailed discussion of hyperparameter choices and good ablations.\n- Paper is well written and easy to follow.\n\nCons:\n- Limited novelty of methods. Ideas similar to the model variants discussed in this work have been considered in other work (Eg: [1]). It is also known that in-domain pre-training (i.e, pre-training on data close to the downstream task’s data distribution) helps (Eg: [2]). So this work can be considered as an application of existing ideas to dialog tasks. \n- In terms of impact, utterance retrieval has fairly limited applicability in dialog. The dialog tasks considered in this work have a maximum of 100 candidate utterances, whereas in practice, the space of possible responses is much larger. While retrieval models are useful, I am skeptical about the practical value of the improvements shown in the paper (especially the improvements over bi-encoder, which is already a decent model).\n\nSuggestions:\nOne way to get around the inefficiency of the cross-encoder architecture is to first use an inexpensive scoring mechanism such as TFIDF or bi-encoder to identify a small number of promising candidates from all the possible candidates. We can then use the cross-encoder to do more precise scoring of only the promising candidates. I am curious where a pipelined model such as this compares against the variants discussed in the paper in terms of speed and performance. \n\nWhile the paper presents strong results on several dialog utterance retrieval tasks, the methods presented have limited novelty and impact. I am hence leaning towards borderline. \n\nReferences\n\n[1] Logeswaran Lajanugen, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-Shot Entity Linking by Reading Entity Descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.\n[2] Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.\n\nEdit: I have read the author response. Based on the rebuttal, I am more convinced about the practical impact of the approach. I am raising my score and recommending accept. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new architecture that achieves the advantages of both Bi-encoder and Cross-encoder architectures. The proposed idea is reasonable and well-motivated, and the paper is clearly written. The experimental results on retrieval and dialog tasks are strong, achieving high accuracy while the computational efficiency is orders of magnitude smaller than Cross-encoder. All reviewers recommend acceptance of the paper and this AC concurs.
Decision: Accept (Poster)