End-to-End Autoregressive Retrieval via Bootstrapping for Smart Reply Systems
Authors: Benjamin Towle
Ke Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: dialog
smart reply
reply suggestion
TL;DR: We present an end-to-end autoregressive retrieval model for the smart reply task, trained via bootstrapping.

Abstract: Reply suggestion systems represent a staple component of many instant messaging and email systems. However, the requirement to produce sets of replies, rather than individual replies, makes the task poorly suited for out-of-the-box retrieval architectures, which only consider individual message-reply similarity. As a result, these system often rely on additional post-processing modules to diversify the outputs. However, these approaches are ultimately bottlenecked by the performance of the initial retriever, which in practice struggles to present a sufficiently diverse range of options to the downstream diversification module, leading to the suggestions being less relevant to the user. In this paper, we consider a novel approach that radically simplifies this pipeline through an autoregressive text-to-text retrieval model, that learns the smart reply task end-to-end from a dataset of (message, reply set) pairs obtained via bootstrapping. Empirical results show this method consistently outperforms a range of state-of-the-art baselines across three datasets, corresponding to a 5.1\%-17.9\% improvement in relevance, and a 0.5\%-63.1\% improvement in diversity compared to the best baseline approach. We make our code publicly available.

End-to-End Autoregressive Retrieval via Bootstrapping for Smart Reply Systems
Benjamin Towle', Ke Zhou 1.2
University of Nottingham
Nokia Bell Labs
(benjamin, towle,,,, ke. zhou ( eeeeeettnnnnnnn ac
Abstract
present diverse set of options the user. For instance, given the message How are you?, an SR system could suggest: {I' m good; Ok; Not great). Resultantly, the quality of a given reply depends not only on the message, but on the other replies in the reply set.
Reply suggestion systems represent staple component of many instant messaging and email systems. However, the requirement to produce of replies, rather than individual replies, makes the task poorly suited for out- of-the-box retrieval architectures, which only consider individual message-rephy similarity. As result, these system often rely on addi- tional post-procsssing modules diversify the outputs. However, these approaches ulti- mately bottlenecked by the performance the initial retriever, which in practice struggles to present sufficiently diverse range of options to the downstream diversification module, lead- ing the suggestions being less relevant to the user. In this paper, we consider novel approach that radically simplifies this pipeline through an autoregressive text-to-text retrieval model, that learns the smart reply task end-to- end from dataset of (message, reply set) pairs obtained via bootstrapping. Empirical results show this method consistently outperforms range of state-of-the-ar baselines across three datasets. corresponding to 91-179992 im- provement in relevance, and 0.5%-63. 1% provement in diversity compared to the best baseline approach. We make our code publicly available.
Several prior works explore solutions to this problem such as removing near duplicates, pe- nalising inter-reply similarity (Deb et al., 2019), clustering by intent (Henderson et al., 2017; Weng et al., 2019), learning latent variables (Deb et al., 2019, 2021), model-based simulation (Towle and Zhou, 2023). However, these methods share a common design choice (Figure 1A): (1) a retrieval- based Matching model, which has learned a shared embedding space between messages and replies, returns a shortlist of top scoring replies; (2) this shortlist is refined through some diversification pro- cedure to obtain the final reply set.
Unfortunately. this assumes that the initial short- list contains at least one good reply set. In practice, we find Matching models often search myopically, only retrieving candidates that are very similar to one another (Figure 1A). Thus, the chosen reply set often fails to reflect diverse range of user intents, while latency constraints make more sophisticated diversification techniques or larger shortlists pro- hibitive (Deb et al., 2019).
1 Introduction
An intuitive, but the best of our knowledge unexplored, solution this problem to con- duct the retrieval autoregressively. with each reply conditioned on both the initial message and the previous replies in the set. Unfortunately, this ap- proach encounters a second problem, namely, the lack of any datasets containing (message, reply set) pairs (Towle and Zhou, 2023). In practice, SR systems are trained individual (message, reply) pairs obtained from conversation datasets, while the task of presenting multiple diverse replies the user is outsourced to a separate diversification module.
Reply suggestion, or smart reply (SR), systems are a staple component of many commercial applica- tions such as Gmail, Skype. Outlook, Microsoft Teams, Linkedln and Facebook Messenger. They help the user process chats and emails quicker by offering a set of canned replies which can be clicked without requiring manual typing. However, dialogue is known be a one-to-many problem (Zhao al., 2017: Towle and Zhou, 2022) nnanel, for any given message, there are multiple possible replies. To reflect this uncertainty, systems should
'https://g/ thwb.com/ www.eeeeeeeeeeeeeeeeeeaeeeae

---

Retrieve N replies
[A] Previous methods
How about this one?
Yensiaga
Diversification (M-CVAE, MMR, SimSR, etc.)
How about this one? its the intest desgg.
Matching model
Whht about this one?
It's a perfect At:
Autaregressive decoding
[8] STAR (Our method)
Xessage design.
Encoder-Decoder
What about this one?
NLL loss
Looks good. IV toke
End-to-end backpropagation
Figure I: Previous methods [A] compared to our approach, STAR [B]. The example displayed is taken from the DailyDialgg Test set, and compares the predictions of STAR with SimSR (Towle and Zhou, 2023), the next best method. Our method's suggestions present a diverse range of topies/intents to drive the conversation.
To meet this dual need, present both (i) bootstrapping method for creating high-quality dataset of (message, reply sets) and (ii) a novel autoregressive retrieval model which predicts se- quences of replies. For solving (i), we observe how model-based planning algorithms have been known to serve as powerful policy improvement operator (Silver et al., 2017; Schrittwieser et al.. 2019), including in several NLP systems (Jang al., 2020, 2021). Specifically, the outputs of model-based planning algorithm can be used to bootstrap a SR system. Further, by conducting this planning offline we are able to leverage two key advantages: (1) the system is free of the latency constraints of online inference, and therefore can increase the search space coverage of the planning algorithm;; (2) the system can leverage information that would not be available during inference, such the ground-truth reply, further guide the search process. For (ii) we unify both steps of the stan- dard SR pipeline into a single end-to-end model, which mitigates the myopic search, and allows the model to learn to diversify its predictions in a prin- cipled way through gradient-based learning. To this end, we present STAR (Suggested replies with T5 and Autoregressive Retrieval) (Figure 1B). At high level, STAR is text-to-text model trained output sequences of replies, where each reply is conditioned both on the initial message and the previous replics in the sequence. Concretely, we in- stantiate our method with the T5 pretrained model (Raffel et al., 2020). We expand T5's vocabulary by
treating each reply in the candidate pool as novel token, and demonstrate simple-yet-effective tech- nique for initialising the new token embeddings, which leverages the model's existing semantic pri- ors. Notably, by treating each reply as token, we limit the number of autoregressive decoding steps required, keeping the model's I's efficiency compara- ble to other retrieval-basee methods.
Empirically, we evaluate our approach on three benchmarks: Reddit (Zhang et al., 2021), which is the only publicly-available SR benchmark, as well as PersonaChat (Zhang et al., 2018) and DailyDi- alog (Li al., 2017) which are both widely-used in dialogue research more broadly (Zhang et al., 2019; Roller al., 2020, inter alia), and share similar conversational style with SR apps. We demonstrate superior performance over state-of- the-art art baselines across datasets, corresponding to 5. 11%%%.9%% improvement in relevance, and a 0.5 15%%63119% improvement in diversity compared to the best baseline approach. We further show comparable efficiency to previous methods, and perform range of ablations motivate design choices.
In summary, our key contributions are as follows: (1) an autoregressive retrieval architccture for quentially predicting suggested replies: (2) boot- strapping framework for generating high-quality data of (message, reply set) pairs; (3) detailed anal- ysis model behaviour and performance including a case study and ablation of key components.

---

2
Related Work
quences of replies in differentiable way.
Bootstrapping The idea of bootstrapping train- ing data from limited resources has received sig- nificant recent interest in NLP, given the newly demonstrated few zero-shot capabilities of many large language models (Brown et al., 2020). It has seen usage in few-shot shot text-classification (Schick and Schütze, 2021a), semantic similarity (Schick and Schütze, 2021b), tool-usage (Schick et al., 2023), retrieval (Izacard and Grave, 2021), se- quence generation (He et al., 2020), and instruction- tuning (Honovich et al., 2023; Wang et al., 2023; Taori et al., 2023), amongst others. These tech- niques can also be seen distillation (Hinton et al., 2015), except that the training typically involves predicting the exact ken targets, rather than using the soft probabilities of a tcacher model. Although sometimes these techniques are used as an addition to supervised learning (He et al., 2020), in our case there are no datasets containing the ideal reply sets to suggest to the user. Instead, we must bootstrap this in a more unsupervised way, by transforming dataset of (message, reply) pairs into a dataset of (message. reply set) pairs.
Smart reply The proprietary nature of data from email and chat applications has led several previous works to use publicly-availabee dialogue datasets (Zhang et al., 2021; Deb et al., 2021; Towle and Zhou, 2023) to benchmark SR methods, due to their analogous conversational nature. While early SR systems used generative models (Kannan et al., 2016), current production systems favour retrieval methods due to their greater controllability of out- puts and superior latency (Deb et al., 2019). In- creasing the diversity of reply suggestions is key focus of previous work, which has been attempted by: (1) mapping replies discrete intents topics (Kannan et al., 2016; Chakravarthi and Pasternack, 2017; Weng et al., 2019); (2) re-wcighting replics according to their similarity with other replies in the set (Carbonell and Goldstein-Stewark, 1998; Deb et al., 2019); (3) learning continuous latent variables to generate multiple queries (Zhao et al., 2017: Deb et al., 2019); (4) using model-based simulation to iteratively search and evaluate the relevance of candidate reply sets (Towle and Zhou, 2023). Our proposed method differs from all of these approaches in that our model learns to ac- count for the interdependencies between replies through end-to-end backpropagation.
form of knowledge
Methodology
In this section, we first describe the model-based planning process used to obtain the bootstrapped dataset of (message, reply set) pairs (Section 3. I). Then, we show how the STAR architecture can be trained on this dataset (Section 3.2).
Autoregressive retrieval Integrating neural re- trieval into the well-established paradigm of text- to-text models is of growing interest. Earlier work focuses on outputting a document ID given query (Tay et al., 2022). Further work has extended this by considering alternate ways of representing the document IDs, such as through unique substrings (Bevilacqua et al., 2022). Another line of work has used autoregressive retrieval for the entity link- ing task (Cao et al., 2021a,b,c), There, the mo- tivation to reduce the large number of entities by relying on the text-to-text model's 's pre-existing vocabulary, rather than having to retrieve embed- dings from a memory-intensive dense index. Our proposed method differs considerably from these previous works both in instantiation and motivation. Iestantiationwwiss we generate multiple replies
3.1 Offline Dataset Creation
Our goal is to transform a dialogue dataset D {xx, y)} of (message, reply) tuples, into a dataset De = {<,Y)) where Y is the set of replies {uk) K to be presented to the user. Algorithm summarises this process. While general to any arbitrary planning algorithm, we choose to instantiate our approach with a modified version of SimSR (Towle and Zhou, 2023), a re- cently released publicly available state-of-the-art SR method, that employs model-based simulation to predict reply sets. As the original algorithm was designed for online inference, make several changes to benefit the offline nature of our version, and detail the full implementation below.
method is
critical to making this possible is the novel boot- strapping technique for creating the dataset of (mes- sage, reply set) pairs to train on. Motivation-wise our goal to be able to condition each reply on both the input message and previous replics in the set, enabling the model to learn to predict se-
The initial retrieval conducted by a Matching model & that separately encodes messages and replies into a shared latent space. Given an encoded

---

Algorithm 1 Offline Dataset Creation. We use N=100, MM1000, a=0.75 and x=0.05 our default setting.
let Yg be the set of currently selected replies, such that initially Y: 0. Then, for each of yn, E Yy. we compute the expected similarity for the union of YG and yn, termed G = Yg U yn for brevity:
Input Matching model o message v, precomputed reply vectors [ye]*t 2 number of candidates N. number of simula- tions M, final reply set size query augmentation coefficient a, redundancy penalty
M Σ J(Y Um um)g(ymlle)
Output reply set Yk
x.y < ((z). Ply)
X < ox + (1 oly
query augmentation
We repeat this process for K timesteps, each time appending the highest scoring reply Ya, i.e. until |Yal = K. Note that this greedy search process implicitly canonicalises the order of the replies, as selecting replics this way causes them to be roughly ordered by individual message-reply rele- vance.
Yv + N-argmax(x yr)
Ys + M-argmax(x ye)
((um/z) x expx Ym
b softmax over top-M scores
Ya + 0 for k - to K do
M
y/ argmax Σ
fY88 ym )a(())( Af(Ya,
YG Yo yy
3.1.1 Adjustments
end for
Yx + Yo return YK
Scaling N and M The original SimSR algorithm was used only in an online setting (Towle and Zhou, 2023). Therefore, the size of the search parameters N (number of replies the shortlist) and M (num- ber of simulated user replies) is kept low (15 and 25 respectively in the original paper). As we only need to run this model offline however to obtain the dataset, we find setting N and M to much larger values improves relevance (we use 100 for both), enabling both broader search (i.e. by increasing N) and more accurate similarity function (i.e. by increasing M).
message x ((x),, retrieves the top N candi- dates from a pool of pre-computed reply vectors YR = [yr) R by combining their dot product simi- larity with a pre-compueed language-modes bias a standard component of SR systems to downweight overly specific replies (Deb et al., 2019).
YN= N-argmax(x yv + (y.))
(1)
We then output the K-tuple Y E (K) that has the highest expected similarity with the human reply, according to some similarity function f(,.)-
Redundancy penalty Early testing showed that scaling the search parameters reduced diversity. We therefore introduce redundancy penalty, which penalises the model for selecting replies that are similar replies already in the set YG. This is anal- ogous to the inter-document similarity penalty used in the maximum marginal relevance IR (informa- tion retrieval) technique (Carbonell and Goldstein- Stewart, 1998).
argmax 2))
(2)
Given the objective in SR is for at least one of the replies to be relevant, the similarity function is defined as a maximum over the sampled reply and each of the replies in the reply set, using term-level F1-score: max F1 (2). y).
We assume is sampled from the ground-truth hu- man distribution p(-( As we do not have access the true human distribution in practice, we in- stead use the same Matching model @ as a proxy for this, given is trained on (message, reply) pairs. We then approximate the expectation by marginal- ising over the top-M most likely replies:
Σ 5088. ym um) Af(Ya, yo)
Query augmentation Unlike during online infer- ence, we also have access to the ground-truth reply when constructing the dataset. Previous work has found that models obtain greater representational capabilities when given access to posterior infor- mation (Paranjape et al., 2022; Towle and Zhou, 2022). We therefore use an augmented query retrieve with the Matching model. This is obtained by interpolating between the message and ground- truth reply embeddings. This biases the model's predictions towards the observed ground-truie -tunt
M JYes Fm (g((ma))))))
argmax
(3)
In practice, it is intractable to evaluate every pos- sible reply tuple, due to their combinatorial seal- ing. We therefore approximate this by greedily con- structing the reply set one reply at a time. Formally,

---

the dataset, while still allowing it benefit from its own learned distribution.
of replies and messages:
K
(x) (1-o)9(y)
(6)
DNLL
log pluule,, Bo...,
(8)
3.2 Proposed STAR Model
4 Experimental Setup
We initialise STAR with T5-based text-to-text language model, which has previously been shown to be effective in autoregressive retrieval (Tay et al., 2022). While some autoregressive retrieval ap- proaches identify their documents/replies through unique substrings (Bevilacqua et al., 2022) or con- strained beam search (Cao et al., 202 we focus approaches requiring only limited number of autoregressive steps, to maintain competitive infer- ence speeds to existing retrieval methods (Section 5.3). There are several alternatives for this such as treating each reply set as a unique token, or sepa- rately training on each (message, reply pair), but ultimately we opted for autoregressively treating each reply as unique token in the vocabulary in order to exploit the compositionality of reply sets (Section 5.2 for performance comparison). Note that the types of replies used in smart reply are usually quite short and concise, e.g. 'how are you', 'I'm fine thanks', 'yes, that's right' etc., systems in deployment only need to retrieve from a pool of 30k or so replies (Deb et al., 2019), in order to pro- vide good coverage of possible user intents. As result, we are able to keep the size of the vocabulary reasonable. Thus, our Winkens U W Wrepiess An obvious challenge this approach that by treating each reply as previ- ously unseen word, it removes any semantic priors the model might have about their meaning. To mit- igate this, we employ bag-of-words initialisation strategy. Hence, we define the embedding of the i- th reply E(yn) as the average over the embeddings of the individual words within Wn. E Ve.
4.1 Baselines
Previous work has largely been closed- source and therefore unavailable for direct comparison (Hen- derson et al., 2017; Weng et al., 2019; Deb et al., 2019). With the exception of SimSR, which has publicly available code 2, we re-implement vari- cty of methods that cover the broad range of pre- vious techniques. Due to its comparable size, all baselines apart from Seq2Seq are initialised with DistilBERT as the encoder backbone. These are summarised as follows:
Seq2Seq is generative encoder-decoder. While current production systems and the majority of lated works use only retrieval models (Deb et al., 2019; Towle and Zhou, 2023), at least one related work includes a standard generative transformer as a baseline (Zhang et al., 2021), which we fol- low here. For maximum comparability with our method. we use the same t - 505-mmall model backbone. For each message, sample K sponses independently.
vocabulary is defined as:
Matching represents the out-of-the-box encoder with no additional diversification strategy and was used as a baseline method by Zhang al. (2021). simply selects the top K responses according individual message-reply scores.
Matching-Topic uses an out-of-the-box topic classifier to ensure no two replics share the same topic, similar to previous work (Henderson et al., 2017; Weng et al., 2019). The classifier trained on Twitter (Antypas et al., 2022), due to their comparable short-form open-domain chat conver- sations.
1 N N
E(yc)
Σ E(w.)
Intuitively, this ensures that the initial embeddings are close to the word embeddings of the original vocabulary, while also capturing some of the under- lying semantics of the reply. We allow the weights update during fine-tuning. Note that for T5 the output and input embedding layers share weights, and therefore this approach is used to initialise both layers. We train the model using cross entsopy loss to predict the next reply given the current sequence
Maximum Marginal Relevance (MMR) (Car- bonell and Goldstein-Stewarr, 1998) is originally an IR technique, used in several previous SR works (Deb et al.. 2019; Towle and Zhou, 2023), which re-weights reply scores linear combination of their message-reply and inter-reply similarity.
Phttps: http::///gghhhbcoooo/oe Towll nowl h/w.w//wwswle/

---

Train
Valid
Test
YR
PersonaChat (Zhang et al., 2018) is crowdworker-soureed dataset comprising persona- grounded conversations, in which each speaker is assigned a persona comprising a few short sentences. Following previous methods (Humeau et al., 2020), we concatenate the persona to the beginning of the message. The participants are instructed to chat naturally and to try to get to know one another.
Reddit
50k
5k
5k
48k
PersonaChat
66k
8k
8k
64k
DailyDialog
76k
7k
7k
62k
Table I: Number of samples in the Train, Validation, Test sets and Candidate pool in the three datasets for evaluation. The Candidate pool comprises the Train set with duplicate responses removed.
MCVAE (Deb al., 2019) is conditional varia- tional autoencoder (Zhao et al., 2017) which learns to generate multiple query vectors from single message embedding, representing the multiple pos- sible reply intents. Candidates are scored via a vot- ing process, whereby the K mest-selected replies are chosen.
DailyDialog et al., 2017) is a dataset created from English language learning websites and con- sists of a variety of high-quality dialogues in every- day scenarios. The dataset differs from the former two in that the conversations often involve real-life scenarios, such as asking for directions, and there- fore captures different variety of conversational skills.
SimSR (Towle and Zhou, 2023) uses an itera- tive search and evaluation process to select possi- ble reply sets and score them according to their expected similarity from a learned world model, which serves as a proxy for the user. To ensure comparability of SimSR with our method and the other baselines, we include the language-modee bias in the scoring process (Equation 1), and also deduplicate the candidate pool.²
4.3 Metries
We evaluate our method the same weighted ROUGE ensemble as previous methods (Lin, 2004; Deb et al., 2019, 2021), which is known correlate well with click-through rate (Zhang et al., 2021):
ROUGE-1 6
ROUGE-2
ROUGE-3
+
(9)
As the goal of SR systems it to ensure that at least one of the suggested replies relevant to the user, we only record the maximum ROUGE score across each the K 3 suggested replics. We also evaluate the model on Self-ROUGE (Celikyilmaz et al., 2020): This is an unreferenced metric that measures the internal dissimilarity (i.e. diversity) within the reply set by treating one reply as the predicted reply and the other parts as the refer- ences. Note that a lower Self-ROUGB score indi- cates more diversity.
4.2 Datasets
We evaluate our proposed method across three datasets, summarised Table 1. Below, we de- scribe the datasets in more detail and motivate their inclusion. Note, other than Reddit, there are no publicly available SR datasets, due to their com- mercial nature (c.g. Henderson et al. (2017); Deb al. (2019); Weng al. (2019)). Therefore, adopt several dialogue datasets, which the clos- est alternative conversations on proprietary chat applications.
4.4 Inference
For inference, we use the entire training set the candidate pool for each respective dataset, with deduplication to remove exact matches. For STAR, we greedily decode the next reply token until tokens have been decoded. Note, we only allow the model to output replies represented in the boot- strapped dataset, and also block non-replies, i.c. words from the original vocabulary, from being predicted.
Reddit (Zhang et al., 2021) was originally intro- duced for training multilingual SR systems, and is the only publicly available dataset specifically intended for SR purposes. As the original dataset is very large, we follow Towle and Zhou (2023) and use the reduced version of the dataset. Note, this version only contains English, as our aim limited to the monolingual setting. Due to the or- ganic nature of the dataset, conversations cover very broad range of topics.
Experimental Results
We focus our efforts on answering the following Research Questions: (RQ1) ) How does STAR com-
Booth changes lead to consistently improved accuracy and diversity across all datascts compared to the original paper.

---

pare existing state-of-the-art methods? (Sec- tion 5.1, 5.4); (RQ2) Which components of the data collection algorithm and fine-tuning have the largest impact on STAR's 's performance? (Section 5.2); (RQs) How efficient is STAR in inference? (Section 5.3)
(C) we replace the bag-of-words embeddings with randomly initialised embeddings this removes any priors about the meaning of replies and forces the model to learn them tabula rasa; (D) we treat each reply set as a unique token - this removes the compositional element from the task, constraining the model to only predicting previously seen reply sets, therefore testing whether the model is capable of learning to compose novel reply sets; (E) we remove the ability to account for interdependencies between replies, by restructuring each (message, reply set) data point into K data points (message, reply;), and then outputting the top- replies dur- ing inference - this investigates whether the benefit lies simply in the bootstrapped dataset being better suited to the SR task, rather than in STAR's 's ability to account for interdependencies between replies.
5.1 Main Results
Table 2 compares the performance of different SR systems across the Reddit, PersonaChat and Dialy- Dialog datasets. In terms of relevance (ROUGE),
STAR shows an especially large improvement in Reddit (+17.9%) and DailyDialog (+15.8%). We hypothesise the gains in PersonaChat (+5.1%) are more modest because the replics are more casily predicted due to the persona, which is concatenated to each message. This significantly reduces the noise during the initial retrieval for the baselines, they only need to retrieve the messages relevant to that particular persona.
In terms of data collection ablations, we found removing the redundancy penalty significantly duced the diversity of predictions, although in some cases offered slightly improved relevance; remov- ing the query augmentation generally led to a worse relevance/diversity trade-off. For the variants of STAR training, we found that random embeddings consistently reduced relevance, while also led to less diverse predictions; reply sets tokens led to the most competitive variant of STAR compared to our default setup: diversity was overall better, due to using preconstructed reply sets from the offline planning algorithm, but this came at the trade-off of reduced flexibility from being unable to con- struct novel reply sets when the context required - resuuaaltn,yy, we saw a corresponding reduction in relevance. Finally, predicting replies separately expectedly harmed both relevance and diversity, demonstrating the importance of accounting for reply interdependencies.
For diversity (Self-ROUGE), the strongest gains were found in DailyDialog (+6..1%) For Per- sonaChat, STAR performs much better than the retrieval methods, only falling behind Seq2Seq, due to its altogether noisier outputs as evidenced by having the worst relevance score. The Reddit results were comparatively more modest (+0.5%) hypothesise this is because the dataset is al- together more noisy, and so there are relatively few similar replies in the dataset, as shown by the Self-ROUGE scores being lower than the other two datasets. Overall, the consistent outperformance in both relevance and diversity metrics supports the benefits of the STAR approach.
5.2 Ablation
In Table 3, we conduct ablations across two key axes: data collection and STAR training. The data collection ablations serve to investigate the benefits of the novel changes to the SimSR algorithm from Section 1. The STAR training ablations inves- tigates the degree to which the improvements in performance are caused by the bootstrapped dataset or by STAR's 's architecture itself; we achieve this by considering several alternative variants of STAR. Our data collection ablations consider two features: (A) removing the query augmentation prevents the model from leveraging any ground truth informa- tion during prediction; (B) removing the redun- dancy penalty no longer explicitly penalises lack of diversity in predicted reply sets. For STAR train- ing, we consider three alternative configurations:
In Figure 2, we further validated the individual results of our ablation by aggregating the results across datasets (applying an equal weighting to each dataset). This demonstrates the overall trend that the default STAR offers the superior trade-off between relevance and diversity, while treating re- ply sets as tokens offered the next best alternative. Nevertheless, we believe that keeping individual replies tokens thus allowing the model con- struct reply sets dynamically - is likely to be an attractive property for deployed systems, enabling the overall vocabulary size to remain modest.

---

Reddit
PersonaChat
Daily Dialog
Method
ROUGE
Self
ROUGE t J ROUGET 1 Self-ROUGE
Generative models
Seq2Seq
2.11
3,43
6.83
6.88*
4.01
3.91
Retrieval models
Matching
1.95
9.42
7.51
21.47
6.53
16.65
M-Topic
1.81
3,94
7.16
15.43
6.14
11.11
M-MMR
2.20
4.44
7.81
14.57
6.13
8.63
M-CVAE
2.30
5.02
7.43
12.21
6.78
10.49
SimSR
2.79
2.18
9.04
10.52
6.82
4.80
STAR
3.29
2.17
9.50
7.74
7.90
1.77
Table 2: Performance of STAR across Reddit, PersonaChat and DailyDialog Test sets relevance (ROUGE) and diversity (Self-ROUGE) metrics. Bold indicates best result, underline indicates second-best. statistically significant versus best result t-test with p-value 0.01.
Model
Reddit
PersonaChat
Daily
Configuration
ROUGE
Relf-
ROUGE
Self-ROUGE
ROUGE
Self-ROUGE
STAR
3.35
2.27
8.85
7.48
8.39
1.81
Data Collection Ablations
A: No Query Augmentation
2.94
2.00
8.99
6.94
7.24
2.89
B: No Redundancy Penalty
3.06
4.29
9.03
17.26
8,98*
5.90
STAR Training Variants
C: Random embeddings
2.67
4.93
8.39
10.97
6.84
4.45
D: Reply sets as tokens
2.85
1.59
8.76
7.75
1.57
E: Predict replics separately
2.20
26.61
8.07
30.98
6.43
20.50
Table 3: Performance of STAR on the Reddit, PersonaChat and DailyDialog Validation sets under different model configurations. Ablations are applied separately. Bold indicates best result, underline indicates second-best. statistically significant versus next best result on t-test with p-value < 0.01.
5.3 Run-time Efficiency
5.4 Case Study
Beyond performance gains in relevance and diver- sity, a major advantage of an autoregressive re- trieval model is the ability to leverage the scalabil- ity of GPU-based inference. Figure compares the efficiency of STAR with the other baseline methods. We use an NVIDIA GeForce RTX 3060 Ti GPU and AMD Ryzen 7 5700G with Radeon Graphics CPU, with a batch size of 32. The results show that the methods can be broadly clustered into three groups. The slowest group is the gen- erative method Seq2Seq, due to needing to gener- ate each reply word-by-wordd The middle group SimSR, M-CVAE and M-MMR is characterised by methods that comprise more involved diversifi- cation pipeline. The final and fastest group includes STAR, M-Topic and Matching, where no additional post-hoc diversification is required (for M-Topic the topics can be pre-computed prior to inference).
Table presents case study the DailyDialog Test set. We compare our approach, STAR, with the top-performing baseline from Table 2, SimSR. In both examples we consistently find STAR able to output a broader range of intents. Quantitatively, we consider the rank that each suggestion receives according to the initial retrieval of the Matching model that underlies SimSR. We see that STAR is able perform much more global search across the reply space, selecting replies from within the top 100 or so ranks. This would difficult for the standard retrieve-and-rerank approach to em- ulate, given 100 is usually too large a number to efficiently rerank (Deb et al., 2019). Qualitatively, SimSR's suggestions converge around common phrases, e.g. 'Iet's go', which would be difficult deduplicate with a heuristic rule given only limited number of overlapping words between the replies. Conversely, STAR able to represent a broader range of intents, such as replying with a question in both examples. Further examples are
"Results surpass reported numbers original paper due to inclusion of language-mode model hias and deduplicated candidate pool to support better comparability (Section 4.

---

ROUGE THHt-OOUF
Sait Sodie
STAR
At Query
B: No Redundancy
Ci Random
Di Reply sets
Predict replies
Figure 2: Comparison of overall relevance and diversity scores across ablations, obtained by averaging across all three datasets with equal weighting.
provided in Appendix C.
6 Conclusion
Seq25eq
We introduce STAR, an autoregressive retrieval system for SR, which is an end-to-end text-to-text model that sequentially predicts replics conditioned on initial message. To train STAR, we demon- strate approach to bootstrap dataset of high- quality (message, reply set) pairs, from regular di- alogue datasets containing only (message, reply) pairs. Empirically, our results show significant im- provement over existing state-of-the-at SR base- lines, across multiple datasets, corresponding to a 5. 111.9% improvement in relevance, and a 0.5%-63.1 improvement in diversity compared to the best baseline approach.
SimSR
M-CVAE
M-MMR
Matching
50
100
150
200
250
300
350
400
Messages/ Second
Figure 3: Comparison of run-time efficiency between STAR and the baseline methods. Results calculated over the Reddit Validation set.
Future work could extend these techniques to other set- set-prediction tasks: e.g., in IR the relevance of each document depends on the quantity new formation it contains compared to other documents in the set. In recommender systems, use cases in- clude: tailoring a user's news feed requires that the news articles presented are not simply dupli- cates of the same story; designing bespoke music playlist requires songs be unified by common themes but also sufficiently distinct from one an- other to maintain the listener's interest. Other lines of future work include considering alternate strate- gies for initialising the reply embeddings, beyond the bag-of-word initialisation demonstrated in this paper.
Message:
Hi, Kenny. Let's go for drink.
let's go [#9]
SimSR
ok let's go. [43]
ok let's get something to drink. [#1]
ok. let's go [#S]
STAR
you want something to drink ? [#89]
good idea. [#105]
Message:
course Ler go.
lltts go [#1]
SimSR
let's go [#5]
all right let's go [#12]
let's go [#1]
STAR
where ? [#43]
good idea [#85]
Acknowledgements
Table 4: Example model outputs from the DailyDi- alog Test set, comparing STAR (ours) with the top- performing baseline method. Numbers bold indicate the ranking the reply received according to the Matching model.
We thank the reviewers for their helpful feedback and suggestions. This work is partly supported by the EPSRC DTP Studentship program. The opinions expressed in this paper are the authors', and are not necessarily shared/endorsed by their employers and/or sponsors.

---

Limitations
criminative correction. In Conference on Empirical Methods in Natural Language Processing.
Although our work shows that STAR is able to absorb sufficient information about the replies in its weights, this may become increasingly chal- lenging when larger numbers of replies need to be embedded. One notable instance of this would be the multilingual setting, as many SR systems are deployed globally. In this case, each language typically has its own candidate pool. A naive imple- mentation which creates separate reply vectors for each language would incur a significant increase in model size. In this case, we hypothesise techniques around weight-sharing between reply embeddings between languages may be beneficial, c.g. 'how are you' (en) and "ça va' (fr) sharing the same vec- tor. Further, our techniques are only demonstrated in publicly available datasets, whereas proprietary conversations in chat and email applications may have unique features not accounted for here (e.g. timestamps, сc and bec information, and file at- tachments). Our technique also requires planning algorithm to create the initial dataset. This the- oretically creates upper bound to the overall performance of STAR, it is limited to cloning the behaviour of the offline planning algorithm.
Nicola De Cao. Gautier Izacard, Sebastian Riedel, and Fabio Petroni. b02 Autoregressive entity retrieval. In 9th International Conference on Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Nicola De Cao, Ledell Yu Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke 7ettle- moyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. 2021c. Multilingual autoregressive entity link- ing. Trunsactions of the Association for Computational Linguistics, 10:274-290.
Jaime G. Carbonell and Jade Goldstein-Stewart, 1998. The use of mmr, diversity- based reranking for reorder- ing documents and producing summaries. In Aunual International ACM SIGIR Conference on Research and Development in Information Retrieval.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: survey. ArXiv, abs/2006. 14799.
Nimesh Chakravarthi and Jeff Pasternack. 2017. Building smart replies for member messages, press re- lease. https: / b1 bllcg/211/100//// Iding- smart smart-replies ies- for member aeeamber-easages. measages.
Budhaditya Deb, Peter Bailey, and Milad Shokouhi. 2019. Diversifying reply suggestions using a matching- conditional variational autoencoder. In North American Chapter of the Association for Computational Linguis- tics.
References
Dimosthenis Antypas. Asahi Ushio, Jose Camacho- Collados, Vitor Silva, Leonardo Neves, and Francesco Barbieri. 2022. Twitter topic classification. In Proceed- ings of the 29th International Conference Computa- tional Linguistics. pages 3386-3400, Gyeongju, Repub- lic of Korea. International Committee on Computational Linguistics.
Budhaditya Deb, Guoqing Zheng, Milad Shokouhi, and Ahmed Hassan Awadallah. 2021. A conditional genera- tive matching model for multi-lingual lingual reply suggestion. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 1553-1568, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Wen tau Yih, Sehastian Riedel, and Fabio Petroni, 2022. Autoregressive search engines: Generating substrings document identifiers. In NeurlPS.
Junxian He. Jiatao Gu, Jiajun Shen, and Mare Aurelio Ranzato. 2020. Revisiting self-training for neural se- quence generation. In Proceedings of ICLR.
Matthew L. Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukáes, Ruiqi Guo, Sanjiv Ku- mar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. CoRR, 0bs/1705.0005)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Asso- ciates, Inc.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. ArXiv, abs/1 503.0253..
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61 Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), pages 14409-14428. Toronto, Canada, Association for Com- putational Linguistics.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021a. Highly parallel autoregressive entity linking with dis-

---

Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux. and Jason Weston. 2020. Poly- ooly-encoders: Architectures and pre-training strategies for fast and accurate multi- sentence scoring. In International Conference on Learn- ing Reprexentations.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer. Nicola Can- cedda, and Thomas Scialom, 2023. Toolformer: Lan- guage models can teach themselves to use tools. ArXiv, abs/2302.04761.
Gautier Izacard and Edouard Grave. 2021. Distilling knowledge from reader to retriever for question answer- ing. In International Conference on Learning Represen- tations.
Timo Schick and Hinrich Schüitze. 202 Exploiting cloze-questions for few-shot text classification and nat- ural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255-269, Online. Association for Computational Lin- guistics.
Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. 2020. monte-carlo planning and learning for goal coil-triented dialogues. In AAAI Conference on Artifi- cial Intelligence.
Timo Schick and Hinrich Schüitze. 2021b. Generating datasets with pretrained language models. In Proceed- ings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 6943-6951, On- line and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Youngsoo Jang. Seokin Seo, Jongmin Lee, and Kee- Eung Kim. 2021. Monte-carlo planning and learning with language action value estimates. In International Conference on Learning Representations.
Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kauf- mann, Andrew Tomkins, Balint Miklos, Gregory S. Cor- rado, László Lukács, Marina Ganea, Peter Young, and Vivek Ramavajjala. 2016. Smart reply: Automated re- sponse suggestion for email. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
Julian Schrittwieser, Ioannis Antonoglou. Thomas Hu- bert, Karen Simonyan, L. Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Grae- pel. Timothy P. Lillicrap, and David Silver. 2019. Mas- tering atari, go, chess and shogi by planning with a learned model. Nature, 588:604- 609.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao. and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In International Joint Conference on Natural Language Processing.
David Silver. Thomas Hubert, Julian Schrittwieser, Ioan- nis Antonoglou, Matthew Lai, Arthur Guez. Marc Lanc- tot, L.. Sifre, Dharshan Kumaran, Thore Graepel, Tim- othy P. Lillicrap, Karen Simonyan, and Demis Hass- abis. 2017. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. ArXiv, abs/1712.018
Chin-Yew Lin. 2004. Rouge: package for automatic evaluation of summaries. In Annual Meeting of the Association for Computational Linguistics.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instructionn-ollooiing llama model. https:// g githubb.cm//eee tato//tttsuuuaaa lab/ stanford_apaca.
Nly Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Ashwin Paranjape, Omar Khattab, Christopher Potts, Matci Zaharia, and Christopher D Manning. 2022. Hind- sight: Posterior-guidee training of retrievers for im- proved open-ended generation. In International Confer- ence on Learning Representations.
Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. Transformer memory as differentiable search index. In Advances in Neural Information Pro- cessing Systems.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(1).
Benjamin Towle and Ke Zhou. 2022. Learn what is possible, then choose what best: Disentangling one- to-many relations in language through text-based games. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 4955 4965,, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Stephen Roller, Emily Dinam, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric Michael Smith, Y.-Lan Boureau, and Jason Weston. 2020. Recipes for building an open-domain chatbot. In Conference of the European Chapter of the Association for Computational Linguistics.
Benjamin Towle and Ke Zhou. 2023. Model-basee sim- ulation for optimising smart reply. In Proceedings of the 61 Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), pages 12030-12043, Toronto, Canada, Association for Com- putational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.

---

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language mod- els with self- generated instructions. In Proceedings of the 6lst Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Pupers), pages 13484-13508, Toronto, Canada. Association for Com- putational Linguistics.
users have been shown be slightly more positive in the sentiment of their emails when shown sug- gested replies (Wenker, 2023). SR systems are on the whole known to produce more positive senti- ment messages than the human distribution (Kan- nan et al., 2016). We see this as an extension of the broader trend of LLMs to be overly obsequious.
Yue Weng, Huaixiu Zheng. Franziska Bell, and Gökhan Tür. 2019. Occ: A smart reply system for efficient in- app communications. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Dis- covery & Data Mining.
B Implementation Details
For constructing the training dataset, we use the fol- lowing hyperparameters: SimSR is initialised from the dist inttille bberrt base messe-une unceeedd checkpoint (Sanh et al., 2019). We set the search parameters to N = 100 and M = 100. We use redundancy penalty of 0.05 and blending alpha of 0.75 for query augmentation. Both parameters provided a good tradc-off between relevance and diversity in early testing, so we did not search hyperparameters further (see Section 5.2 for ablations).
Kilian Wenker. 2023. Who wrote this? how smart replies impact language and agency in the workplace. Telematics and Informatics Reports, 10: 10:100062.
Mozhi Zhang, Wei Wang, Budhaditya Deb, Guoqing Zheng, Milad Shokouhi, and Ahmed Hassan Awadallah. 2021. A dataset and baselines for multilingual reply suggestion, In Proceedings of the 59th Anmual Meet- ing of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pages 1207-1220. Online. Association for Computational Lin- guistics.
For training STAR, we initialise our model with the t55-mall checkpoint. Note that this version of T5 has a comparable parameter count to the base- lines which use DistilBERT (60M versus 66M). We tokenise the dataset with a maximum message length of 64 tokens. We train our model for up 100k steps, with warmup of 1 steps. In practice, the model typically converged around 20k steps. We use the A optimiser (Loshchilov and Hut- ter, 2019) 9) with an initial learning rate of 5e-4 and linear decay. We evaluate every 2k steps, by taking the ROUGE and Relf-ROUGE scores on the valida- tion set (this uses the ground-truth from the original dataset, not from the bootstrapped dataset), and em- ploy early stopping once both metrics have ceased to improve. Note, we found these metries were a much more reliable stopping point than crossen- tropy loss, which typically converged much earlier.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur D. Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? In Annual Meeting of the Association for Computational Linguistics.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao. Jianfeng Gao, Jingiin Liu, and William B. Dolan. 2019. Dialogpt Large-scale generative pre-training for conversational response gen- eration. In Annual Meeting of the Association for Com- putational Linguistics.
Tiancheng Zhao, Ran Zhao, and Maxine Eskénazi. 2017. Learning discourse-level diversity for neural dialog mod- els using conditional variational autoencoders. In ACI.
A Ethical Considerations
Controlling the outputs of dialogue models is fore- front issue in ethics research for AI, particularly with the impact of recent gains in LLM capabilities. We believe the risks in the case of SR systems have several mitigants compared to this: the replies can be vetted by humans before deployment; the replies are usually in short-form, rather containing com- plex information the user may rely on; ultimately, a user must seleet one of the options, rather than the system being able to reply without user oversight. Conversely, there are some risks more unique to SR systems that should be mentioned. Particularly, the suggestions presented by the system can have subtle priming effects on user behaviour. Notably,
C Further Case Studies
Table 5 displays further examples of STAR's pre- dictions versus SimSR, taken from the PersonaChat Test set.

---

my favorite tv show is the office.
like jazz music.
Persona:
do not drive because live in new york. i am jewish.
do not eat anything sweet.
Message:
i was but now divorced
i am sorry to hear that
SimSR
oh i am sorry to hear that
i am sorry to hear that do you have any kids ?
oh i am sorry to hear that
STAR
do you like music ? like music ? love jazz
hey there my name is jordan and am veterinarian.
love read drama books.
Persona:
love playing video games. am also a musician on the weekends.
am originally from california but live florida.
Message:
sometimes. listen to a lot of music. do you read a lot
hi how are you? ?
SimSR
hi how are you today
i do not like music at all
i listen to music do you no not really like all music
STAR
yes do my favorite is country
Table Example model outputs from the PersonaChat Test set, comparing STAR (ours) with the top-performing baseline method. STAR able capture broader range of intents through its end-to-end autoregressive retrieval.


Reviews:
[{"review_id": "UyLJkQIzPo", "reviewer": "Reviewer_YiYZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the helpful feedback!\n- We will adjust the methodology explanation to make it more self-contained to avoid requiring readers to be familiar with SimSR. There is a typo in the pseudocode, as x tilde should be used to obtain Y_N and Y_M instead of x.\n- We made two changes to SimSR to align it with our method for fair comparison: (i) we use the language model bias (equation 1) which is not used in original SimSR paper; (ii) we deduplicate the candidate pool (Y_R) as some basic replies, e.g. 'how are you?', occurred multiple times. These design choices closer align both methods with reported industry techniques [1]. We emphasise that these changes improved SimSR's overall performance versus the original version (e.g. ROUGE Reddit 2.40 -> 2.79, ROUGE PersonaChat 7.71 -> 9.04), and that STAR is still significantly better than the original unadjusted results from the SimSR paper. We will update the paper to make these adjustments clearer.\n- When training STAR on the newly created dataset, we use the ground-truth reply from the original dataset to calculate the ROUGE (lines 883ff.), whereby we take the maximum ROUGE between the ground-truth reply and each of the three predicted replies. We use this setup during validation as the task of predicting at least one relevant reply converged at a different rate to the task of predicting the sequence of replies from the newly created dataset. As our final evaluation setup is concerned with the former task (at least one relevant reply) we also use this to validate the model's performance during training. We will make this clearer in the paper.\n- We will update Section 5.2 to include a reference to Table 3. We find the results are consistent across all three datasets (as in the case of the main results). Due to limited space, we restricted the ablation and case study to only showing a single dataset each. We would use the additional space to expand this to all three datasets.\n\n[1] Deb et al., 2019. Diversifying reply suggestions using a matching-conditional variational autoencoder. NAACL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* There are enough experiments to answer ``How does STAR compare to existing state-of-the-art methods?'', ``Which components of the data collection algorithm and fine-tuning have the largest impact on STAR’s performance?''  and ``How efficient is STAR in inference?''.\nIn addition, the author provides much discussion about the experiment results and gives reasonable explanation of several phenomena.\n\n* The bootstrapping data construction method can get high-quality data of pairs  and improves the performance of SR task. ", "reasons_to_reject": "* The symbols in Algorithm 1 are confusing, for example,  “x tilde”  is used to represent the augmented query, but it is not used in the entire pseudocode, so how does it work?  How are Y_N and Y_M used together to select the final K replies, I have to read the paper of SimSR to figure it out.\n\n* Why the results of SimSR in Table 2 is different from that in its original paper? Since SimSR has released its source code, what caused this difference?\n\n* For each message in the newly created dataset, there are multiple replies. So how is the ROUGE calculated, one reference or multiple references?\n\n* No references to Table 3 in Section 5.2. Why Reddit is chosen to perform ablation while DialyDialog is chosen for the case study?", "questions_to_authors": "", "paper_topic_and_main_contributions": "To address the less relevance problem of reply suggestion from previous SR works, the author first constructed a new dataset based on the original SR dataset and trained the STAR model on the new dataset. Then, STAR based on T4 was proposed to treat each reply as a new token, and generates multiple replies in an autoregressive manner. Experimental results show that the proposed method improves both the diversity and relevance on three datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZOeE5EygjK", "reviewer": "Reviewer_RPm6", "summary": "", "questions": "The results of SimSR presented in Table 2 differ from those in Table 3 of the original SimSR paper. Could you please clarify the differences between your re-implementations and the original paper's findings?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the helpful feedback!\n- We note that the results are statistically significant versus the other methods (on t-test with p-value < 0.01). The absolute value of ROUGE tends to be small in open-domain tasks such as this due to the large number of possible replies for a given message [1]. Unfortunately we do not have a deployed smart reply system to gauge click-through rate to obtain subjective evaluation, however we draw confidence from the fact that the ROUGE metrics used are reported to be well-correlated with click-through rate by previous research [1].\n- We recognise that the baselines for M-Topic, M-MMR and M-CVAE are reimplementations, due to the original methods not being publicly-available; however, for SimSR, which is the strongest-performing baseline, we use the publicly-available code. We believe the baselines selected represent the range of techniques used in Smart Reply, and there are no other publicly-available techniques that we have omitted. We will release the code of both our method and the reimplementations to enable reproducibility. \n- While in this instance the quality of our model is limited by the performance of SimSR, our STAR method is generic and can be used with any bootstrapping model. We therefore believe that discovering new bootstrapping models is an opportunity for future research. We will update the paper to reflect this.\n- We made two changes to SimSR to align it with our method for fair comparison: (i) we use the language model bias (equation 1) which is not used in original SimSR paper; (ii) we deduplicate the candidate pool (Y_R) as some basic replies, e.g. 'how are you?', occurred multiple times. These design choices closer align both methods with reported industry techniques [2]. We emphasise that these changes improved SimSR's overall performance versus the original version (e.g. ROUGE Reddit 2.40 -> 2.79, ROUGE PersonaChat 7.71 -> 9.04), and that STAR is still significantly better than the original unadjusted results from the SimSR paper. We will update the paper to make these adjustments clearer.\n\n[1] Zhang et al., 2021. A dataset and baselines for multilingual reply suggestion. ACL.\n\n[2] Deb et al., 2019. Diversifying reply suggestions using a matching-conditional variational autoencoder. NAACL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method stands out from previous approaches, which relied on matching and diversification models. Instead, the new method features an autoregressive text-to-text retrieval approach that operates in an end-to-end manner.\n2. The motivation for the proposed method is transparent and the approach is both straightforward and effective. Compared to the baselines, the new method outperforms them in both relevance and diversity, while also demonstrating competitive run-time efficiency when compared to the fastest matching methods available.", "reasons_to_reject": "1. As there is no subjective evaluation and the absolute value of ROUGE is very small, the objective evaluation alone is insufficient for accurately comparing and evaluating different models.\n2. It is worth noting that all baselines were re-implemented by the authors, which may introduce some degree of unfairness when compared to previous methods.\n3. The proposed method requires a SimSR model for bootstrapping the dataset, and as such, the performance may be limited by the upperbound of SimSR's performance.", "questions_to_authors": "The results of SimSR presented in Table 2 differ from those in Table 3 of the original SimSR paper. Could you please clarify the differences between your re-implementations and the original paper's findings?", "paper_topic_and_main_contributions": "This paper presents an autoregressive text-to-text retrieval model designed to learn the smart reply task from a dataset of (message, reply set) pairs obtained through bootstrapping. The experimental results demonstrate that this approach outperforms the baselines in both relevance and diversity across three datasets. The main contributions of the paper are: (1) proposing an autoregressive retrieval method for sequentially predicting suggested replies, and (2) introducing a bootstrapping framework for generating (message, reply set) pairs from (message, reply) data.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "21vI2wDpng", "reviewer": "Reviewer_86dA", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the helpful feedback!\n- We note that inference latency is actually lower compared to many other methods, as their handcrafted diversification techniques often impose a larger latency cost. We report these findings in Figure 2.\n- As Smart Reply systems typically cover a more limited range of intents (e.g. 'how are you', 'I'm fine thanks', 'yes, that's right' etc.) compared to fully open-domain dialogue systems, systems in deployment only need to retrieve from a pool of 30k or so replies [1], in order to provide good coverage of possible user intents. As a result, we are able to keep the size of the vocabulary reasonable. We will update the paper to include that this may be a limitation for transferring the techniques to some tasks outside of Smart Reply if they require a much larger vocabulary than this, although we believe most tasks should fall within this range.\n- Although bag-of-words embeddings outperformed random initialisation (Table 3), we recognise there may be more sophisticated initialisation techniques. We believe this is an interesting area for future work, and will update the paper to reflect this.\n- Our central motivation for combining these discrete techniques is to obtain as strong as possible a dataset in order to train our STAR model, which is one of the main contributions of this work (We note that STAR is generic and can be trained on any dataset with a <message, reply set> structure). We do this by taking advantage of the unique benefits of offline inference--lack of latency constraints and availability of ground-truth reply. Lastly, the redundancy penalty is attempting to counteract overly similar replies, which empirically (Table 3) we found to be a negative side-effect of the other two adjustments. We will update the paper to better make these motivations clear.\n\n[1] Deb et al., 2019. Diversifying reply suggestions using a matching-conditional variational autoencoder. NAACL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The issue of chosen reply set failing to reflect a diverse range of user intents is worth investigation.\n\n(2) Experimental results on three benchmarks demonstrate improvement in relevance and diversity.", "reasons_to_reject": "(1) Conditioning each reply on the previous replies in the set sequentially might increase the inference latency.\n\n(2) \"Expand T5’s vocabulary by treating each reply in the candidate pool as a novel token\": the augmented vocabulary might be huge due to the perplexity of sentences.\n\n(3) bag-of-words initialisation strategy for averaging over the embeddings of the individual words might not be a good choice for representing replies.\n\n(4) The adjustments in section 3.1.1 just combined several existing and discrete techniques for improving performance, which do not address a core research issue.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents an autoregressive retrieval method to diversify the reply set for smart reply system.\nSpecifically, retrieval is conducted with each reply conditioned on both the initial message and the previous replies in the set. \nSeveral techniques are adopted to address the subsequent issues in this architecture, such as scaling, and redundancy penalty.\nExperimental results on three benchmarks demonstrate improvement in relevance and diversity.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel autoregressive text-to-text retrieval model designed to learn the smart reply task from a dataset using a bootstrapping method for generating data.  Extensive experiments demonstrate superior performance of the proposed method over baselines in terms of relevance and diversity.  The reviewers requested some clarifications in their reviews, which were addressed by the authors in their rebuttal.  Another concern raised was the absence of a subjective (human) evaluation, which the authors addressed to some extent by pointing to evidence of correlation between ROUGE and click-through rate.  However, the addition of a subjective human judgement evaluation of relevance and diversity might have been useful.
Decision: Accept-Findings