Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring

1 Introduction Recently , substantial improvements to state-of-the-art benchmarks on a variety of language understanding tasks have been achieved through the use of deep pre-trained language models followed by fine-tuning ( Devlin et al. , 2019 ) . In this work we explore improvements to this approach for the class of tasks that require multi-sentence scoring : given an input context , score a set of candidate labels , a setup common in retrieval and dialogue tasks , amongst others . Performance in such tasks has to be measured via two axes : prediction quality and prediction speed , as scoring many candidates can be prohibitively slow . The current state-of-the-art focuses on using BERT models for pre-training ( Devlin et al. , 2019 ) , which employ large text corpora on general subjects : Wikipedia and the Toronto Books Corpus ( Zhu et al. , 2015 ) . Two classes of fine-tuned architecture are typically built on top : Bi-encoders and Cross-encoders . Cross-encoders ( Wolf et al. , 2019 ; Vig & Ramea , 2019 ) , which perform full ( cross ) self-attention over a given input and label candidate , tend to attain much higher accuracies than their counterparts , Bi-encoders ( Mazaré et al. , 2018 ; Dinan et al. , 2019 ) , which perform self-attention over the input and candidate label separately and combine them at the end for a final representation . As the representations are separate , Bi-encoders are able to cache the encoded candidates , and reuse these representations for each input resulting in fast prediction times . Cross-encoders must recompute the encoding for each input and label ; as a result , they are prohibitively slow at test time . In this work , we provide novel contributions that improve both the quality and speed axes over the current state-of-the-art . We introduce the Poly-encoder , an architecture with an additional learnt attention mechanism that represents more global features from which to perform self-attention , resulting in performance gains over Bi-encoders and large speed gains over Cross-Encoders . To pre-train our architectures , we show that choosing abundant data more similar to our downstream task also brings significant gains over BERT pre-training . This is true across all different architecture choices and downstream tasks we try . We conduct experiments comparing the new approaches , in addition to analysis of what works best for various setups of existing methods , on four existing datasets in the domains of dialogue and information retrieval ( IR ) , with pre-training strategies based on Reddit ( Mazaré et al. , 2018 ) compared ∗ Joint First Authors . to Wikipedia/Toronto Books ( i.e. , BERT ) . We obtain a new state-of-the-art on all four datasets with our best architectures and pre-training strategies , as well as providing practical implementations for real-time use . Our code and models will be released open-source . 2 RelatedWork The task of scoring candidate labels given an input context is a classical problem in machine learning . While multi-class classification is a special case , the more general task involves candidates as structured objects rather than discrete classes ; in this work we consider the inputs and the candidate labels to be sequences of text . There is a broad class of models that map the input and a candidate label separately into a common feature space wherein typically a dot product , cosine or ( parameterized ) non-linearity is used to measure their similarity . We refer to these models as Bi-encoders . Such methods include vector space models ( Salton et al. , 1975 ) , LSI ( Deerwester et al. , 1990 ) , supervised embeddings ( Bai et al. , 2009 ; Wu et al. , 2018 ) and classical siamese networks ( Bromley et al. , 1994 ) . For the next utterance prediction tasks we consider in this work , several Bi-encoder neural approaches have been considered , in particular Memory Networks ( Zhang et al. , 2018a ) and Transformer Memory networks ( Dinan et al. , 2019 ) as well as LSTMs ( Lowe et al. , 2015 ) and CNNs ( Kadlec et al. , 2015 ) which encode input and candidate label separately . A major advantage of Bi-encoder methods is their ability to cache the representations of a large , fixed candidate set . Since the candidate encodings are independent of the input , Bi-encoders are very efficient during evaluation . Researchers have also studied a more rich class of models we refer to as Cross-encoders , which make no assumptions on the similarity scoring function between input and candidate label . Instead , the concatenation of the input and a candidate serve as a new input to a nonlinear function that scores their match based on any dependencies it wants . This has been explored with Sequential Matching Network CNN-based architectures ( Wu et al. , 2017 ) , Deep Matching Networks ( Yang et al. , 2018 ) , Gated Self-Attention ( Zhang et al. , 2018b ) , and most recently transformers ( Wolf et al. , 2019 ; Vig & Ramea , 2019 ; Urbanek et al. , 2019 ) . For the latter , concatenating the two sequences of text results in applying self-attention at every layer . This yields rich interactions between the input context and the candidate , as every word in the candidate label can attend to every word in the input context , and vice-versa . Urbanek et al . ( 2019 ) employed pre-trained BERT models , and fine-tuned both Bi- and Cross-encoders , explicitly comparing them on dialogue and action tasks , and finding that Cross-encoders perform better . However , the performance gains come at a steep computational cost . Cross-encoder representations are much slower to compute , rendering some applications infeasible . 3 Tasks We consider the tasks of sentence selection in dialogue and article search in IR . The former is a task extensively studied and recently featured in two competitions : the Neurips ConvAI2 competition ( Dinan et al. , 2020 ) , and the DSTC7 challenge , Track 1 ( Yoshino et al. , 2019 ; Jonathan K. Kummerfeld & Lasecki , 2018 ; Chulaka Gunasekara & Lasecki , 2019 ) . We compare on those two tasks and in addition , we also test on the popular Ubuntu V2 corpus ( Lowe et al. , 2015 ) . For IR , we use the Wikipedia Article Search task of Wu et al . ( 2018 ) . The ConvAI2 task is based on the Persona-Chat dataset ( Zhang et al. , 2018a ) which involves dialogues between pairs of speakers . Each speaker is given a persona , which is a few sentences that describe a character they will imitate , e.g . “ I love romantic movies ” , and is instructed to get to know the other . Models should then condition their chosen response on the dialogue history and the lines of persona . As an automatic metric in the competition , for each response , the model has to pick the correct annotated utterance from a set of 20 choices , where the remaining 19 were other randomly chosen utterances from the evaluation set . Note that in a final system however , one would retrieve from the entire training set of over 100k utterances , but this is avoided for speed reasons in common evaluation setups . The best performing competitor out of 23 entrants in this task achieved 80.7 % accuracy on the test set utilizing a pre-trained Transformer fine-tuned for this task ( Wolf et al. , 2019 ) . The DSTC7 challenge ( Track 1 ) consists of conversations extracted from Ubuntu chat logs , where one partner receives technical support for various Ubuntu-related problems from the other . The best performing competitor ( with 20 entrants in Track 1 ) in this task achieved 64.5 % R @ 1 ( Chen & Wang , 2019 ) . Ubuntu V2 is a similar but larger popular corpus , created before the competition ( Lowe et al. , 2015 ) ; we report results for this dataset as well , as there are many existing results on it . Finally , we evaluate on Wikipedia Article Search ( Wu et al. , 2018 ) . Using the 2016-12-21 dump of English Wikipedia ( ∼5M articles ) , the task is given a sentence from an article as a search query , find the article it came from . Evaluation ranks the true article ( minus the sentence ) against 10,000 other articles using retrieval metrics . This mimics a web search like scenario where one would like to search for the most relevant articles ( web documents ) . The best reported method is the learningto-rank embedding model , StarSpace , which outperforms fastText , SVMs , and other baselines . We summarize all four datasets and their statistics in Table 1 . 4 Methods In this section we describe the various models and methods that we explored . 4.1 Transformers and Pre-training Strategies Transformers Our Bi- , Cross- , and Poly-encoders , described in sections 4.2 , 4.3 and 4.4 respectively , are based on large pre-trained transformer models with the same architecture and dimension as BERT-base ( Devlin et al. , 2019 ) , which has 12 layers , 12 attention heads , and a hidden size of 768 . As well as considering the BERT pre-trained weights , we also explore our own pre-training schemes . Specifically , we pre-train two more transformers from scratch using the exact same architecture as BERT-base . One uses a similar training setup as in BERT-base , training on 150 million of examples of [ INPUT , LABEL ] extracted from Wikipedia and the Toronto Books Corpus , while the other is trained on 174 million examples of [ INPUT , LABEL ] extracted from the online platform Reddit ( Mazaré et al. , 2018 ) , which is a dataset more adapted to dialogue . The former is performed to verify that reproducing a BERT-like setting gives us the same results as reported previously , while the latter tests whether pre-training on data more similar to the downstream tasks of interest helps . For training both new setups we used XLM ( Lample & Conneau , 2019 ) . Input Representation Our pre-training input is the concatenation of input and label [ INPUT , LABEL ] , where both are surrounded with the special token [ S ] , following Lample & Conneau ( 2019 ) . When pre-training on Reddit , the input is the context , and the label is the next utterance . When pre-training on Wikipedia and Toronto Books , as in Devlin et al . ( 2019 ) , the input is one sentence and the label the next sentence in the text . Each input token is represented as the sum of three embeddings : the token embedding , the position ( in the sequence ) embedding and the segment embedding . Segments for input tokens are 0 , and for label tokens are 1 . Pre-training Procedure Our pre-training strategy involves training with a masked language model ( MLM ) task identical to the one in Devlin et al . ( 2019 ) . In the pre-training on Wikipedia and Toronto Books we add a next-sentence prediction task identical to BERT training . In the pre-training on Reddit , we add a next-utterance prediction task , which is slightly different from the previous one as an utterance can be composed of several sentences . During training 50 % of the time the candidate is the actual next sentence/utterance and 50 % of the time it is a sentence/utterance randomly taken from the dataset . We alternate between batches of the MLM task and the next-sentence/nextutterance prediction task . Like in Lample & Conneau ( 2019 ) we use the Adam optimizer with learning rate of 2e-4 , β1 = 0.9 , β2 = 0.98 , no L2 weight decay , linear learning rate warmup , and inverse square root decay of the learning rate . We use a dropout probability of 0.1 on all layers , and a batch of 32000 tokens composed of concatenations [ INPUT , LABEL ] with similar lengths . We train the model on 32 GPUs for 14 days . Fine-tuning After pre-training , one can then fine-tune for the multi-sentence selection task of choice , in our case one of the four tasks from Section 3 . We consider three architectures with which we fine-tune the transformer : the Bi-encoder , Cross-encoder and newly proposed Poly-encoder . 4.2 Bi-encoder In a Bi-encoder , both the input context and the candidate label are encoded into vectors : yctxt = red ( T1 ( ctxt ) ) ycand = red ( T2 ( cand ) ) where T1 and T2 are two transformers that have been pre-trained following the procedure described in 4.1 ; they initially start with the same weights , but are allowed to update separately during finetuning . T ( x ) = h1 , .. , hN is the output of a transformer T and red ( · ) is a function that reduces that sequence of vectors into one vector . As the input and the label are encoded separately , segment tokens are 0 for both . To resemble what is done during our pre-training , both the input and label are surrounded by the special token [ S ] and therefore h1 corresponds to [ S ] . We considered three ways of reducing the output into one representation via red ( · ) : choose the first output of the transformer ( corresponding to the special token [ S ] ) , compute the average over all outputs or the average over the first m ≤ N outputs . We compare them in Table 7 in the Appendix . We use the first output of the transformer in our experiments as it gives slightly better results . Scoring The score of a candidate candi is given by the dot-product s ( ctxt , candi ) = yctxt ·ycandi . The network is trained to minimize a cross-entropy loss in which the logits are yctxt ·ycand1 , ... , yctxt ·ycandn , where cand1 is the correct label and the others are chosen from the training set . Similar to Mazaré et al . ( 2018 ) , during training we consider the other labels in the batch as negatives . This allows for much faster training , as we can reuse the embeddings computed for each candidate , and also use a larger batch size ; e.g. , in our experiments on ConvAI2 , we were able to use batches of 512 elements . Inference speed In the setting of retrieval over known candidates , a Bi-encoder allows for the precomputation of the embeddings of all possible candidates of the system . After the context embedding yctxt is computed , the only operation remaining is a dot product between yctxt and every candidate embedding , which can scale to millions of candidates on a modern GPU , and potentially billions using nearest-neighbor libraries such as FAISS ( Johnson et al. , 2019 ) . 4.3 Cross-encoder The Cross-encoder allows for rich interactions between the input context and candidate label , as they are jointly encoded to obtain a final representation . Similar to the procedure in pre-training , the context and candidate are surrounded by the special token [ S ] and concatenated into a single vector , which is encoded using one transformer . We consider the first output of the transformer as the context-candidate embedding : yctxt , cand = h1 = f irst ( T ( ctxt , cand ) ) where f irst is the function that takes the first vector of the sequence of vectors produced by the transformer . By using a single transformer , the Cross-encoder is able to perform self-attention between the context and candidate , resulting in a richer extraction mechanism than the Bi-encoder . As the candidate label can attend to the input context during the layers of the transformer , the Crossencoder can produce a candidate-sensitive input representation , which the Bi-encoder can not . For example , this allows it to select useful input features per candidate . Scoring To score one candidate , a linear layer W is applied to the embedding yctxt , cand to reduce it from a vector to a scalar : s ( ctxt , candi ) = yctxt , candi W Similarly to what is done for the Bi-encoder , the network is trained to minimize a cross entropy loss where the logits are s ( ctxt , cand1 ) , ... , s ( ctxt , candn ) , where cand1 is the correct candidate and the others are negatives taken from the training set . Unlike in the Bi-encoder , we can not recycle the other labels of the batch as negatives , so we use external negatives provided in the training set . The Cross-encoder uses much more memory than the Bi-encoder , resulting in a much smaller batch size . Inference speed Unfortunately , the Cross-encoder does not allow for precomputation of the candidate embeddings . At inference time , every candidate must be concatenated with the input context and must go through a forward pass of the entire model . Thus , this method can not scale to a large amount of candidates . We discuss this bottleneck further in Section 5.4 . 4.4 Poly-encoder The Poly-encoder architecture aims to get the best of both worlds from the Bi- and Cross-encoder . A given candidate label is represented by one vector as in the Bi-encoder , which allows for caching candidates for fast inference time , while the input context is jointly encoded with the candidate , as in the Cross-encoder , allowing the extraction of more information . The Poly-encoder uses two separate transformers for the context and label like a Bi-encoder , and the candidate is encoded into a single vector ycandi . As such , the Poly-encoder method can be implemented using a precomputed cache of encoded responses . However , the input context , which is typically much longer than a candidate , is represented with m vectors ( y1ctxt .. y m ctxt ) instead of just one as in the Bi-encoder , where m will influence the inference speed . To obtain these m global features that represent the input , we learn m context codes ( c1 , ... , cm ) , where ci extracts representation yictxt by attending over all the outputs of the previous layer . That is , we obtain yictxt using : yictxt = ∑ j wcij h j where ( w ci 1 , .. , w ci N ) = softmax ( ci · h1 , .. , ci · hN ) The m context codes are randomly initialized , and learnt during finetuning . Finally , given our m global context features , we attend over them using ycandi as the query : yctxt = ∑ i wiyictxt where ( w1 , .. , wm ) = softmax ( ycandi · y1ctxt , .. , ycandi · ymctxt ) The final score for that candidate label is then yctxt · ycandi as in a Bi-encoder . As m < N , where N is the number of tokens , and the context-candidate attention is only performed at the top layer , this is far faster than the Cross-encoder ’ s full self-attention . 5 Experiments We perform a variety of experiments to test our model architectures and training strategies over four tasks . For metrics , we measure Recall @ k where each test example has C possible candidates to select from , abbreviated to R @ k/C , as well as mean reciprocal rank ( MRR ) . 5.1 Bi-encoders and Cross-encoders We first investigate fine-tuning the Bi- and Cross-encoder architectures initialized with the weights provided by Devlin et al . ( 2019 ) , studying the choice of other hyperparameters ( we explore our own pre-training schemes in section 5.3 ) . In the case of the Bi-encoder , we can use a large number of negatives by considering the other batch elements as negative training samples , avoiding recomputation of their embeddings . On 8 Nvidia Volta v100 GPUs and using half-precision operations ( i.e . float16 operations ) , we can reach batches of 512 elements on ConvAI2 . Table 2 shows that in this setting , we obtain higher performance with a larger batch size , i.e . more negatives , where 511 negatives yields the best results . For the other tasks , we keep the batch size at 256 , as the longer sequences in those datasets uses more memory . The Cross-encoder is more computationally intensive , as the embeddings for the ( context , candidate ) pair must be recomputed each time . We thus limit its batch size to 16 and provide negatives random samples from the training set . For DSTC7 and Ubuntu V2 , we choose 15 such negatives ; For ConvAI2 , the dataset provides 19 negatives . The above results are reported with Bi-encoder aggregation based on the first output . Choosing the average over all outputs instead is very similar but slightly worse ( 83.1 , averaged over 5 runs ) . We also tried to add further non-linearities instead of the inner product of the two representations , but could not obtain improved results over the simpler architecture ( results not shown ) . We tried two optimizers : Adam ( Kingma & Ba , 2015 ) with weight decay of 0.01 ( as recommended by ( Devlin et al. , 2019 ) ) and Adamax ( Kingma & Ba , 2015 ) without weight decay ; based on validation set performance , we choose to fine-tune with Adam when using the BERT weights . The learning rate is initialized to 5e-5 with a warmup of 100 iterations for Bi- and Poly-encoders , and 1000 iterations for the Cross-encoder . The learning rate decays by a factor of 0.4 upon plateau of the loss evaluated on the valid set every half epoch . In Table 3 we show validation performance when fine-tuning various layers of the weights provided by ( Devlin et al. , 2019 ) , using Adam with decay optimizer . Fine-tuning the entire network is important , with the exception of the word embeddings . With the setups described above , we fine-tune the Bi- and Cross-encoders on the datasets , and report the results in Table 4 . On the first three tasks , our Bi-encoders and Cross-encoders outperform the best existing approaches in the literature when we fine-tune from BERT weights . E.g. , the Biencoder reaches 81.7 % R @ 1 on ConvAI2 and 66.8 % R @ 1 on DSTC7 , while the Cross-encoder achieves higher scores of 84.8 % R @ 1 on ConvAI2 and 67.4 % R @ 1 on DSTC7 . Overall , Crossencoders outperform all previous approaches on the three dialogue tasks , including our Bi-encoders ( as expected ) . We do not report fine-tuning of BERT for Wikipedia IR as we can not guarantee the test set is not part of the pre-training for that dataset . In addition , Cross-encoders are also too slow to evaluate on the evaluation setup of that task , which has 10k candidates . 5.2 Poly-encoders We train the Poly-encoder using the same batch sizes and optimizer choices as in the Bi-encoder experiments . Results are reported in Table 4 for various values of m context vectors . The Poly-encoder outperforms the Bi-encoder on all the tasks , with more codes generally yielding larger improvements . Our recommendation is thus to use as large a code size as compute time allows ( see Sec . 5.4 ) . On DSTC7 , the Poly-encoder architecture with BERT pretraining reaches 68.9 % R1 with 360 intermediate context codes ; this actually outperforms the Cross-encoder result ( 67.4 % ) and is noticeably better than our Bi-encoder result ( 66.8 % ) . Similar conclusions are found on Ubuntu V2 and ConvAI2 , although in the latter Cross-encoders give slightly better results . We note that since reporting our results , the authors of Li et al . ( 2019 ) have conducted a human evaluation study on ConvAI2 , in which our Poly-encoder architecture outperformed all other models compared against , both generative and retrieval based , including the winners of the competition . 5.3 Domain-specific Pre-training We fine-tune our Reddit-pre-trained transformer on all four tasks ; we additionally fine-tune a transformer that was pre-trained on the same datasets as BERT , specifically Toronto Books + Wikipedia . When using our pre-trained weights , we use the Adamax optimizer and optimize all the layers of the transformer including the embeddings . As we do not use weight decay , the weights of the final layer are much larger than those in the final layer of BERT ; to avoid saturation of the attention layer in the Poly-encoder , we re-scaled the last linear layer so that the standard deviation of its output matched that of BERT , which we found necessary to achieve good results . We report results of fine-tuning with our pre-trained weights in Table 4 . We show that pre-training on Reddit gives further state-ofthe-art performance over our previous results with BERT , a finding that we see for all three dialogue tasks , and all three architectures . The results obtained with fine-tuning on our own transformers pre-trained on Toronto Books + Wikipedia are very similar to those obtained with the original BERT weights , indicating that the choice of dataset used to pre-train the models impacts the final results , not some other detail in our training . Indeed , as the two settings pre-train with datasets of similar size , we can conclude that choosing a pre-training task ( e.g . dialogue data ) that is similar to the downstream tasks of interest ( e.g . dialogue ) is a likely explanation for these performance gains , in line with previous results showing multi-tasking with similar tasks is more useful than with dissimilar ones ( Caruana , 1997 ) . 5.4 Inference Speed An important motivation for the Poly-encoder architecture is to achieve better results than the Biencoder while also performing at a reasonable speed . Though the Cross-encoder generally yields strong results , it is prohibitively slow . We perform speed experiments to determine the trade-off of improved performance from the Poly-encoder . Specifically , we predict the next utterance for 100 dialogue examples in the ConvAI2 validation set , where the model scores C candidates ( in this case , chosen from the training set ) . We perform these experiments on both CPU-only and GPU setups . CPU computations were run on an 80 core Intel Xeon processor CPU E5-2698 . GPU computations were run on a single Nvidia Quadro GP100 using cuda 10.0 and cudnn 7.4 . We show the average time per example for each architecture in Table 5 . The difference in timing between the Bi-encoder and the Poly-encoder architectures is rather minimal when there are only 1000 candidates for the model to consider . The difference is more pronounced when considering 100k candidates , a more realistic setup , as we see a 5-6x slowdown for the Poly-encoder variants . Nevertheless , both models are still tractable . The Cross-encoder , however , is 2 orders of magnitude slower than the Bi-encoder and Poly-encoder , rendering it intractable for real-time inference , e.g . when interacting with a dialogue agent , or retrieving from a large set of documents . Thus , Polyencoders , given their desirable performance and speed trade-off , are the preferred method . We additionally report training times in the Appendix , Table 6 . Poly-encoders also have the benefit of being 3-4x faster to train than Cross-encoders ( and are similar in training time to Bi-encoders ) . 6 Conclusion In this paper we present new architectures and pre-training strategies for deep bidirectional transformers in candidate selection tasks . We introduced the Poly-encoder method , which provides a mechanism for attending over the context using the label candidate , while maintaining the ability to precompute each candidate ’ s representation , which allows for fast real-time inference in a production setup , giving an improved trade off between accuracy and speed . We provided an experimental analysis of those trade-offs for Bi- , Poly- and Cross-encoders , showing that Poly-encoders are more accurate than Bi-encoders , while being far faster than Cross-encoders , which are impractical for real-time use . In terms of training these architectures , we showed that pre-training strategies more closely related to the downstream task bring strong improvements . In particular , pre-training from scratch on Reddit allows us to outperform the results we obtain with BERT , a result that holds for all three model architectures and all three dialogue datasets we tried . However , the methods introduced in this work are not specific to dialogue , and can be used for any task where one is scoring a set of candidates , which we showed for an information retrieval task as well . References Bing Bai , Jason Weston , David Grangier , Ronan Collobert , Kunihiko Sadamasa , Yanjun Qi , Olivier Chapelle , and Kilian Weinberger . Supervised semantic indexing . In Proceedings of the 18th ACM conference on Information and knowledge management , pp . 187–196 . ACM , 2009 . Jane Bromley , Isabelle Guyon , Yann LeCun , Eduard Säckinger , and Roopak Shah . Signature verification using a ” siamese ” time delay neural network . In Advances in neural information processing systems , pp . 737–744 , 1994 . Rich Caruana . Multitask learning . Machine learning , 28 ( 1 ) :41–75 , 1997 . Qian Chen and Wen Wang . Sequential attention-based network for noetic end-to-end response selection . CoRR , abs/1901.02609 , 2019 . URL http : //arxiv.org/abs/1901.02609 . Lazaros Polymenakos Chulaka Gunasekara , Jonathan K. Kummerfeld and Walter S. Lasecki . Dstc7 task 1 : Noetic end-to-end response selection . In 7th Edition of the Dialog System Technology Challenges at AAAI 2019 , January 2019 . URL http : //workshop.colips.org/dstc7/ papers/dstc7_task1_final_report.pdf . Scott Deerwester , Susan T Dumais , George W Furnas , Thomas K Landauer , and Richard Harshman . Indexing by latent semantic analysis . Journal of the American society for information science , 41 ( 6 ) :391–407 , 1990 . Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies , Volume 1 ( Long and Short Papers ) , pp . 4171–4186 , Minneapolis , Minnesota , June 2019 . Association for Computational Linguistics . doi : 10.18653/v1/N19-1423 . Emily Dinan , Stephen Roller , Kurt Shuster , Angela Fan , Michael Auli , and Jason Weston . Wizard of Wikipedia : Knowledge-powered conversational agents . In Proceedings of the International Conference on Learning Representations ( ICLR ) , 2019 . Emily Dinan , Varvara Logacheva , Valentin Malykh , Alexander Miller , Kurt Shuster , Jack Urbanek , Douwe Kiela , Arthur Szlam , Iulian Serban , Ryan Lowe , Shrimai Prabhumoye , Alan W. Black , Alexander Rudnicky , Jason Williams , Joelle Pineau , Mikhail Burtsev , and Jason Weston . The second conversational intelligence challenge ( convai2 ) . In Sergio Escalera and Ralf Herbrich ( eds . ) , The NeurIPS ’ 18 Competition , pp . 187–208 , Cham , 2020 . Springer International Publishing . ISBN 978-3-030-29135-8 . Jianxiong Dong and Jim Huang . Enhance word representation for out-of-vocabulary on ubuntu dialogue corpus . CoRR , abs/1802.02614 , 2018 . URL http : //arxiv.org/abs/1802.02614 . Jia-Chen Gu , Zhen-Hua Ling , Yu-Ping Ruan , and Quan Liu . Building sequential inference models for end-to-end response selection . CoRR , abs/1812.00686 , 2018 . URL http : //arxiv.org/ abs/1812.00686 . J. Johnson , M. Douze , and H. Jgou . Billion-scale similarity search with gpus . IEEE Transactions on Big Data , pp . 1–1 , 2019 . ISSN 2372-2096. doi : 10.1109/TBDATA.2019.2921572 . Joseph Peper Vignesh Athreya Chulaka Gunasekara Jatin Ganhotra Siva Sankalp Patel Lazaros Polymenakos Jonathan K. Kummerfeld , Sai R. Gouravajhala and Walter S. Lasecki . Analyzing assumptions in conversation disentanglement research through the lens of a new dataset and model . ArXiv e-prints , October 2018 . URL https : //arxiv.org/pdf/1810.11118.pdf . Rudolf Kadlec , Martin Schmid , and Jan Kleindienst . Improved deep learning baselines for ubuntu corpus dialogs . CoRR , abs/1510.03753 , 2015 . URL http : //arxiv.org/abs/1510.03753 . Diederik P. Kingma and Jimmy Ba . Adam : A method for stochastic optimization . In 3rd International Conference on Learning Representations , ICLR 2015 , San Diego , CA , USA , May 7-9 , 2015 , Conference Track Proceedings , 2015 . URL http : //arxiv.org/abs/1412.6980 . Guillaume Lample and Alexis Conneau . Cross-lingual language model pretraining . Advances in Neural Information Processing Systems ( NeurIPS ) , 2019 . Margaret Li , Jason Weston , and Stephen Roller . Acute-eval : Improved dialogue evaluation with optimized questions and multi-turn comparisons . arXiv preprint arXiv:1909.03087 , 2019 . Ryan Lowe , Nissan Pow , Iulian Serban , and Joelle Pineau . The ubuntu dialogue corpus : A large dataset for research in unstructured multi-turn dialogue systems . In SIGDIAL Conference , 2015 . Pierre-Emmanuel Mazaré , Samuel Humeau , Martin Raison , and Antoine Bordes . Training millions of personalized dialogue agents . In EMNLP , 2018 . Gerard Salton , Anita Wong , and Chung-Shu Yang . A vector space model for automatic indexing . Communications of the ACM , 18 ( 11 ) :613–620 , 1975 . Jack Urbanek , Angela Fan , Siddharth Karamcheti , Saachi Jain , Samuel Humeau , Emily Dinan , Tim Rocktäschel , Douwe Kiela , Arthur Szlam , and Jason Weston . Learning to speak and act in a fantasy text adventure game . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( EMNLP-IJCNLP ) , pp . 673–683 , Hong Kong , China , November 2019 . Association for Computational Linguistics . doi : 10.18653/v1/D19-1062 . Jesse Vig and Kalai Ramea . Comparison of transfer-learning approaches for response selection in multi-turn conversations . Workshop on DSTC7 , 2019 . Thomas Wolf , Victor Sanh , Julien Chaumond , and Clement Delangue . Transfertransfo : A transfer learning approach for neural network based conversational agents . arXiv preprint arXiv:1901.08149 , 2019 . Ledell Yu Wu , Adam Fisch , Sumit Chopra , Keith Adams , Antoine Bordes , and Jason Weston . Starspace : Embed all the things ! In Thirty-Second AAAI Conference on Artificial Intelligence , 2018 . Yu Ping Wu , Wei Chung Wu , Chen Xing , Ming Zhou , and Zhoujun Li . Sequential matching network : A new architecture for multi-turn response selection in retrieval-based chatbots . In ACL , 2017 . Liu Yang , Minghui Qiu , Chen Qu , Jiafeng Guo , Yongfeng Zhang , W Bruce Croft , Jun Huang , and Haiqing Chen . Response ranking with deep matching networks and external knowledge in information-seeking conversation systems . In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , pp . 245–254 . ACM , 2018 . Seunghyun Yoon , Joongbo Shin , and Kyomin Jung . Learning to rank question-answer pairs using hierarchical recurrent encoder with latent topic clustering . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies , Volume 1 ( Long Papers ) , pp . 1575–1584 , New Orleans , Louisiana , June 2018 . Association for Computational Linguistics . doi : 10.18653/v1/N18-1142 . Koichiro Yoshino , Chiori Hori , Julien Perez , Luis Fernando D ’ Haro , Lazaros Polymenakos , R. Chulaka Gunasekara , Walter S. Lasecki , Jonathan K. Kummerfeld , Michel Galley , Chris Brockett , Jianfeng Gao , William B. Dolan , Xiang Gao , Huda AlAmri , Tim K. Marks , Devi Parikh , and Dhruv Batra . Dialog system technology challenge 7 . CoRR , abs/1901.03461 , 2019 . Saizheng Zhang , Emily Dinan , Jack Urbanek , Arthur Szlam , Douwe Kiela , and Jason Weston . Personalizing dialogue agents : I have a dog , do you have pets too ? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics , pp . 2204–2213 , Melbourne , Australia , July 2018a . Association for Computational Linguistics . Zhuosheng Zhang , Jiangtong Li , Pengfei Zhu , Hai Zhao , and Gongshen Liu . Modeling multi-turn conversation with deep utterance aggregation . In COLING , 2018b . Yukun Zhu , Ryan Kiros , Richard S. Zemel , Ruslan R. Salakhutdinov , Raquel Urtasun , Antonio Torralba , and Sanja Fidler . Aligning books and movies : Towards story-like visual explanations by watching movies and reading books . 2015 IEEE International Conference on Computer Vision ( ICCV ) , pp . 19–27 , 2015 . A Training Time We report the training time on 8 GPU Volta 100 for the 3 datasets considered and for 4 types of models in Table 6 . B Reduction layer in Bi-encoder We provide in Table 7 the results obtained for different types of reductions on top of the Bi-encoder . Specifically we compare the Recall @ 1/20 on the ConvAI2 validation set when taking the first output of BERT , the average of the first 16 outputs , the average of the first 64 outputs and all of them except the first one ( [ S ] ) . C Alternative Choices for Context Vectors We considered a few other ways to derive the context vectors ( y1ctxt , ... , y m ctxt ) of the Poly-encoder from the output ( h1ctxt , ... , h N ctxt ) of the underlying transformer : • Learn m codes ( c1 , ... , cm ) , where ci extracts representation yictxt by attending over all the outputs ( h1ctxt , ... , h N ctxt ) . This method is denoted “ Poly-encoder ( Learnt-codes ) ” or “ Poly- encoder ( Learnt-m ) ” , and is the method described in section 4.4 • Consider the first m outputs ( h1ctxt , ... , hmctxt ) . This method is denoted “ Poly-encoder ( First m outputs ) ” or “ Poly-encoder ( First-m ) ” . Note that when N < m , only m vectors are considered . • Consider the last m outputs . • Consider the last m outputs concatenated with the first one , h1ctxt which plays a particular role in BERT as it corresponds to the special token [ S ] . The performance of those four methods is evaluated on the validation set of Convai2 and DSTC7 and reported on Table 8 . The first two methods are shown in Figure 2 . We additionally provide the inference time for a given number of candidates coming from the Convai2 dataset on Table 9 .

Summary: This work proposes a new transformer architecture for tasks that involve a query sequence and multiple candidate sequences. The proposed architecture, called poly-encoder, strikes a balance between a dual encoder which independently encodes the query and candidate and combines representations at the top, and a more expressive architecture which does full joint attention over the concatenated query and candidate sequences. Experiments on utterance retrieval tasks for dialog and an information retrieval task show that poly-encoders strike a good trade-off between the inference speed of the dual encoder model and the performance of the full attention model.