prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Was PolyReponse evaluated against some baseline?
Context 1: Introduction and Background
Task-oriented dialogue systems are primarily designed to search and interact with large databases which contain information pertaining to a certain dialogue domain: the main purpose of such systems is to assist the users in accomplishing a well-defined task such as flight booking BIBREF0, tourist information BIBREF1, restaurant search BIBREF2, or booking a taxi BIBREF3  These systems are typically constructed around rigid task-specific ontologies BIBREF1, BIBREF4 which enumerate the constraints the users can express using a collection of slots (e g , price range for restaurant search) and their slot values (e g , cheap, expensive for the aforementioned slots)  Conversations are then modelled as a sequence of actions that constrain slots to particular values  This explicit semantic space is manually engineered by the system designer  It serves as the output of the natural language understanding component as well as the input to the language generation component both in traditional modular systems BIBREF5, BIBREF6 and in more recent end-to-end task-oriented dialogue systems BIBREF7, BIBREF8, BIBREF9, BIBREF3 
Working with such explicit semantics for task-oriented dialogue systems poses several critical challenges on top of the manual time-consuming domain ontology design  First, it is difficult to collect domain-specific data labelled with explicit semantic representations  As a consequence, despite recent data collection efforts to enable training of task-oriented systems across multiple domains BIBREF0, BIBREF3, annotated datasets are still few and far between, as well as limited in size and the number of domains covered  Second, the current approach constrains the types of dialogue the system can support, resulting in artificial conversations, and breakdowns when the user does not understand what the system can and cannot support  In other words, training a task-based dialogue system for voice-controlled search in a new domain always implies the complex, expensive, and time-consuming process of collecting and annotating sufficient amounts of in-domain dialogue data 
In this paper, we present a demo system based on an alternative approach to task-oriented dialogue  Relying on non-generative response retrieval we describe the PolyResponse conversational search engine and its application in the task of restaurant search and booking  The engine is trained on hundreds of millions of real conversations from a general domain (i e , Reddit), using an implicit representation of semantics that directly optimizes the task at hand  It learns what responses are appropriate in different conversational contexts, and consequently ranks a large pool of responses according to their relevance to the current user utterance and previous dialogue history (i e , dialogue context) 
The technical aspects of the underlying conversational search engine are explained in detail in our recent work BIBREF11, while the details concerning the Reddit training data are also available in another recent publication BIBREF12  In this demo, we put focus on the actual practical usefulness of the search engine by demonstrating its potential in the task of restaurant search, and extending it to deal with multi-modal data  We describe a PolyReponse system that assists the users in finding a relevant restaurant according to their preference, and then additionally helps them to make a booking in the selected restaurant  Due to its retrieval-based design, with the PolyResponse engine there is no need to engineer a structured ontology, or to solve the difficult task of general language generation  This design also bypasses the construction of dedicated decision-making policy modules  The large ranking model already encapsulates a lot of knowledge about natural language and conversational flow 
Since retrieved system responses are presented visually to the user, the PolyResponse restaurant search engine is able to combine text responses with relevant visual information (e g , photos from social media associated with the current restaurant and related to the user utterance), effectively yielding a multi-modal response  This setup of using voice as input, and responding visually is becoming more and more prevalent with the rise of smart screens like Echo Show and even mixed reality  Finally, the PolyResponse restaurant search engine is multilingual: it is currently deployed in 8 languages enabling search over restaurants in 8 cities around the world  System snapshots in four different languages are presented in Figure FIGREF16, while screencast videos that illustrate the dialogue flow with the PolyResponse engine are available at: https://tinyurl com/y3evkcfz 
PolyResponse: Conversational Search
The PolyResponse system is powered by a single large conversational search engine, trained on a large amount of conversational and image data, as shown in Figure FIGREF2  In simple words, it is a ranking model that learns to score conversational replies and images in a given conversational context  The highest-scoring responses are then retrieved as system outputs  The system computes two sets of similarity scores: 1) $S(r,c)$ is the score of a candidate reply $r$ given a conversational context $c$, and 2) $S(p,c)$ is the score of a candidate photo $p$ given a conversational context $c$  These scores are computed as a scaled cosine similarity of a vector that represents the context ($h_c$), and a vector that represents the candidate response: a text reply ($h_r$) or a photo ($h_p$)  For instance, $S(r,c)$ is computed as $S(r,c)=C cos(h_r,h_c)$, where $C$ is a learned constant  The part of the model dealing with text input (i e , obtaining the encodings $h_c$ and $h_r$) follows the architecture introduced recently by Henderson:2019acl  We provide only a brief recap here; see the original paper for further details 
PolyResponse: Conversational Search ::: Text Representation 
The model, implemented as a deep neural network, learns to respond by training on hundreds of millions context-reply $(c,r)$ pairs  First, similar to Henderson:2017arxiv, raw text from both $c$ and $r$ is converted to unigrams and bigrams  All input text is first lower-cased and tokenised, numbers with 5 or more digits get their digits replaced by a wildcard symbol #, while words longer than 16 characters are replaced by a wildcard token LONGWORD  Sentence boundary tokens are added to each sentence  The vocabulary consists of the unigrams that occur at least 10 times in a random 10M subset of the Reddit training set (see Figure FIGREF2) plus the 200K most frequent bigrams in the same random subset 
During training, we obtain $d$-dimensional feature representations ($d=320$) shared between contexts and replies for each unigram and bigram jointly with other neural net parameters  A state-of-the-art architecture based on transformers BIBREF13 is then applied to unigram and bigram vectors separately, which are then averaged to form the final 320-dimensional encoding  That encoding is then passed through three fully-connected non-linear hidden layers of dimensionality $1,024$  The final layer is linear and maps the text into the final $l$-dimensional ($l=512$) representation: $h_c$ and $h_r$  Other standard and more sophisticated encoder models can also be used to provide final encodings $h_c$ and $h_r$, but the current architecture shows a good trade-off between speed and efficacy with strong and robust performance in our empirical evaluations on the response retrieval task using Reddit BIBREF14, OpenSubtitles BIBREF15, and AmazonQA BIBREF16 conversational test data, see BIBREF12 for further details 
In training the constant $C$ is constrained to lie between 0 and $\sqrt{l}$  Following Henderson:2017arxiv, the scoring function in the training objective aims to maximise the similarity score of context-reply pairs that go together, while minimising the score of random pairings: negative examples  Training proceeds via SGD with batches comprising 500 pairs (1 positive and 499 negatives) 
PolyResponse: Conversational Search ::: Photo Representation 
Photos are represented using convolutional neural net (CNN) models pretrained on ImageNet BIBREF17  We use a MobileNet model with a depth multiplier of 1 4, and an input dimension of $224 \times 224$ pixels as in BIBREF18  This provides a $1,280 \times 1 4 = 1,792$-dimensional representation of a photo, which is then passed through a single hidden layer of dimensionality $1,024$ with ReLU activation, before being passed to a hidden layer of dimensionality 512 with no activation to provide the final representation $h_p$ 
PolyResponse: Conversational Search ::: Data Source 1: Reddit 
For training text representations we use a Reddit dataset similar to AlRfou:2016arxiv  Our dataset is large and provides natural conversational structure: all Reddit data from January 2015 to December 2018, available as a public BigQuery dataset, span almost 3 7B comments BIBREF12  We preprocess the dataset to remove uninformative and long comments by retaining only sentences containing more than 8 and less than 128 word tokens  After pairing all comments/contexts $c$ with their replies $r$, we obtain more than 727M context-reply $(c,r)$ pairs for training, see Figure FIGREF2 
PolyResponse: Conversational Search ::: Data Source 2: Yelp 
Once the text encoding sub-networks are trained, a photo encoder is learned on top of a pretrained MobileNet CNN, using data taken from the Yelp Open dataset: it contains around 200K photos and their captions  Training of the multi-modal sub-network then maximises the similarity of captions encoded with the response encoder $h_r$ to the photo representation $h_p$  As a result, we can compute the score of a photo given a context using the cosine similarity of the respective vectors  A photo will be scored highly if it looks like its caption would be a good response to the current context 
PolyResponse: Conversational Search ::: Index of Responses 
The Yelp dataset is used at inference time to provide text and photo candidates to display to the user at each step in the conversation  Our restaurant search is currently deployed separately for each city, and we limit the responses to a given city  For instance, for our English system for Edinburgh we work with 396 restaurants, 4,225 photos (these include additional photos obtained using the Google Places API without captions), 6,725 responses created from the structured information about restaurants that Yelp provides, converted using simple templates to sentences of the form such as “Restaurant X accepts credit cards ”, 125,830 sentences extracted from online reviews 
PolyResponse: Conversational Search ::: PolyResponse in a Nutshell 
The system jointly trains two encoding functions (with shared word embeddings) $f(context)$ and $g(reply)$ which produce encodings $h_c$ and $h_r$, so that the similarity $S(c,r)$ is high for all $(c,r)$ pairs from the Reddit training data and low for random pairs  The encoding function $g()$ is then frozen, and an encoding function $t(photo)$ is learnt which makes the similarity between a photo and its associated caption high for all (photo, caption) pairs from the Yelp dataset, and low for random pairs  $t$ is a CNN pretrained on ImageNet, with a shallow one-layer DNN on top  Given a new context/query, we then provide its encoding $h_c$ by applying $f()$, and find plausible text replies and photo responses according to functions $g()$ and $t()$, respectively  These should be responses that look like answers to the query, and photos that look like they would have captions that would be answers to the provided query 
At inference, finding relevant candidates given a context reduces to computing $h_c$ for the context $c$ , and finding nearby $h_r$ and $h_p$ vectors  The response vectors can all be pre-computed, and the nearest neighbour search can be further optimised using standard libraries such as Faiss BIBREF19 or approximate nearest neighbour retrieval BIBREF20, giving an efficient search that scales to billions of candidate responses 
The system offers both voice and text input and output  Speech-to-text and text-to-speech conversion in the PolyResponse system is currently supported by the off-the-shelf Google Cloud tools 
Dialogue Flow
The ranking model lends itself to the one-shot task of finding the most relevant responses in a given context  However, a restaurant-browsing system needs to support a dialogue flow where the user finds a restaurant, and then asks questions about it  The dialogue state for each search scenario is represented as the set of restaurants that are considered relevant  This starts off as all the restaurants in the given city, and is assumed to monotonically decrease in size as the conversation progresses until the user converges to a single restaurant  A restaurant is only considered valid in the context of a new user input if it has relevant responses corresponding to it  This flow is summarised here:
S1  Initialise $R$ as the set of all restaurants in the city  Given the user's input, rank all the responses in the response pool pertaining to restaurants in $R$ 
S2  Retrieve the top $N$ responses $r_1, r_2, \ldots , r_N$ with corresponding (sorted) cosine similarity scores: $s_1 \ge s_2 \ge \ldots \ge s_N$ 
S3  Compute probability scores $p_i \propto \exp (a \cdot s_i)$ with $\sum _{i=1}^N p_i$, where $a>0$ is a tunable constant 
S4  Compute a score $q_e$ for each restaurant/entity $e \in R$, $q_e = \sum _{i: r_i \in e} p_i$ 
S5  Update $R$ to the smallest set of restaurants with highest $q$ whose $q$-values sum up to more than a predefined threshold $t$ 
S6  Display the most relevant responses associated with the updated $R$, and return to S2 
If there are multiple relevant restaurants, one response is shown from each  When only one restaurant is relevant, the top $N$ responses are all shown, and relevant photos are also displayed  The system does not require dedicated understanding, decision-making, and generation modules, and this dialogue flow does not rely on explicit task-tailored semantics  The set of relevant restaurants is kept internally while the system narrows it down across multiple dialogue turns  A simple set of predefined rules is used to provide a templatic spoken system response: e g , an example rule is “One review of $e$ said $r$”, where $e$ refers to the restaurant, and $r$ to a relevant response associated with $e$  Note that while the demo is currently focused on the restaurant search task, the described “narrowing down” dialogue flow is generic and applicable to a variety of applications dealing with similar entity search 
The system can use a set of intent classifiers to allow resetting the dialogue state, or to activate the separate restaurant booking dialogue flow  These classifiers are briefly discussed in §SECREF4 
Other Functionality ::: Multilinguality 
The PolyResponse restaurant search is currently available in 8 languages and for 8 cities around the world: English (Edinburgh), German (Berlin), Spanish (Madrid), Mandarin (Taipei), Polish (Warsaw), Russian (Moscow), Korean (Seoul), and Serbian (Belgrade)  Selected snapshots are shown in Figure FIGREF16, while we also provide videos demonstrating the use and behaviour of the systems at: https://tinyurl com/y3evkcfz  A simple MT-based translate-to-source approach at inference time is currently used to enable the deployment of the system in other languages: 1) the pool of responses in each language is translated to English by Google Translate beforehand, and pre-computed encodings of their English translations are used as representations of each foreign language response; 2) a provided user utterance (i e , context) is translated to English on-the-fly and its encoding $h_c$ is then learned  We plan to experiment with more sophisticated multilingual models in future work 
Other Functionality ::: Voice-Controlled Menu Search 
An additional functionality enables the user to get parts of the restaurant menu relevant to the current user utterance as responses  This is achieved by performing an additional ranking step of available menu items and retrieving the ones that are semantically relevant to the user utterance using exactly the same methodology as with ranking other responses  An example of this functionality is shown in Figure FIGREF21 
Other Functionality ::: Resetting and Switching to Booking 
The restaurant search system needs to support the discrete actions of restarting the conversation (i e , resetting the set $R$), and should enable transferring to the slot-based table booking flow  This is achieved using two binary intent classifiers, that are run at each step in the dialogue  These classifiers make use of the already-computed $h_c$ vector that represents the user's latest text  A single-layer neural net is learned on top of the 512-dimensional encoding, with a ReLU activation and 100 hidden nodes  To train the classifiers, sets of 20 relevant paraphrases (e g , “Start again”) are provided as positive examples  Finally, when the system successfully switches to the booking scenario, it proceeds to the slot filling task: it aims to extract all the relevant booking information from the user (e g , date, time, number of people to dine)  The entire flow of the system illustrating both the search phase and the booking phase is provided as the supplemental video material 
Conclusion and Future Work
This paper has presented a general approach to search-based dialogue that does not rely on explicit semantic representations such as dialogue acts or slot-value ontologies, and allows for multi-modal responses  In future work, we will extend the current demo system to more tasks and languages, and work with more sophisticated encoders and ranking functions  Besides the initial dialogue flow from this work (§SECREF3), we will also work with more complex flows dealing, e g , with user intent shifts 
Context 2: Introduction
Pre-training of language models has been shown to provide large improvements for a range of language understanding tasks BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   The key idea is to train a large generative model on vast corpora and use the resulting representations on tasks for which only limited amounts of labeled data is available  Pre-training of sequence to sequence models has been previously investigated for text classification BIBREF4 but not for text generation  In neural machine translation, there has been work on transferring representations from high-resource language pairs to low-resource settings BIBREF5  
In this paper, we apply pre-trained representations from language models to language generation tasks that can be modeled by sequence to sequence architectures  Previous work on integrating language models with sequence to sequence models focused on the decoder network and added language model representations right before the output of the decoder BIBREF6   We extend their study by investigating several other strategies such as inputting ELMo-style representations BIBREF0 or fine-tuning the language model (§ SECREF2 ) 
Our experiments rely on strong transformer-based language models trained on up to six billion tokens (§ SECREF3 )  We present a detailed study of various strategies in different simulated labeled training data scenarios and observe the largest improvements in low-resource settings but gains of over 1 BLEU are still possible when five million sentence-pairs are available  The most successful strategy to integrate pre-trained representations is as input to the encoder network (§ SECREF4 ) 
Strategies to add representations
We consider augmenting a standard sequence to sequence model with pre-trained representations following an ELMo-style regime (§ SECREF2 ) as well as by fine-tuning the language model (§ SECREF3 ) 
ELMo augmentation
The ELMo approach of BIBREF0 forms contextualized word embeddings based on language model representations without adjusting the actual language model parameters  Specifically, the ELMo module contains a set of parameters INLINEFORM0 to form a linear combination of the INLINEFORM1 layers of the language model: ELMo = INLINEFORM2 where INLINEFORM3 is a learned scalar, INLINEFORM4 is a constant to normalize the INLINEFORM5 to sum to one and INLINEFORM6 is the output of the INLINEFORM7 -th language model layer; the module also considers the input word embeddings of the language model  We also apply layer normalization BIBREF7 to each INLINEFORM8 before computing ELMo vectors 
We experiment with an ELMo module to input contextualized embeddings either to the encoder () or the decoder ()  This provides word representations specific to the current input sentence and these representations have been trained on much more data than is available for the text generation task 
Fine-tuning approach
Fine-tuning the pre-trained representations adjusts the language model parameters by the learning signal of the end-task BIBREF1 , BIBREF3   We replace learned input word embeddings in the encoder network with the output of the language model ()  Specifically, we use the language model representation of the layer before the softmax and feed it to the encoder  We also add dropout to the language model output  Tuning separate learning rates for the language model and the sequence to sequence model may lead to better performance but we leave this to future work  However, we do tune the number of encoder blocks INLINEFORM0 as we found this important to obtain good accuracy for this setting  We apply the same strategy to the decoder: we input language model representations to the decoder network and fine-tune the language model when training the sequence to sequence model () 
Datasets
We train language models on two languages: One model is estimated on the German newscrawl distributed by WMT'18 comprising 260M sentences or 6B tokens  Another model is trained on the English newscrawl data comprising 193M sentences or 5B tokens  We learn a joint Byte-Pair-Encoding (BPE; Sennrich et al , 2016) vocabulary of 37K types on the German and English newscrawl and train the language models with this vocabulary 
We consider two benchmarks: Most experiments are run on the WMT'18 English-German (en-de) news translation task and we validate our findings on the WMT'18 English-Turkish (en-tr) news task  For WMT'18 English-German, the training corpus consists of all available bitext excluding the ParaCrawl corpus and we remove sentences longer than 250 tokens as well as sentence-pairs with a source/target length ratio exceeding 1 5  This results in 5 18M sentence pairs  We tokenize all data with the Moses tokenizer BIBREF8 and apply the BPE vocabulary learned on the monolingual corpora 
For WMT'18 English-Turkish, we use all of the available bitext comprising 208K sentence-pairs without any filtering  We develop on newstest2017 and test on newstest2018  For en-tr we only experiment with adding representations to the encoder and therefore apply the language model vocabulary to the source side  For the target vocabulary we learn a BPE code with 32K merge operations on the Turkish side of the bitext  Both datasets are evaluated in terms of case-sensitive de-tokenized BLEU BIBREF9 , BIBREF10  
We consider the abstractive document summarization task comprising over 280K news articles paired with multi-sentence summaries  is a widely used dataset for abstractive text summarization  Following BIBREF11 , we report results on the non-anonymized version of rather than the entity-anonymized version BIBREF12 , BIBREF13 because the language model was trained on full text  Articles are truncated to 400 tokens BIBREF11 and we use a BPE vocabulary of 32K types BIBREF14   We evaluate in terms of F1-Rouge, that is Rouge-1, Rouge-2 and Rouge-L BIBREF15  
Language model pre-training
We consider two types of architectures: a bi-directional language model to augment the sequence to sequence encoder and a uni-directional model to augment the decoder  Both use self-attention BIBREF16 and the uni-directional model contains INLINEFORM0 transformer blocks, followed by a word classifier to predict the next word on the right  The bi-directional model solves a cloze-style token prediction task at training time BIBREF17   The model consists of two towers, the forward tower operates left-to-right and the tower operating right-to-left as backward tower; each tower contains INLINEFORM1 transformer blocks  The forward and backward representations are combined via a self-attention module and the output of this module is used to predict the token at position INLINEFORM2   The model has access to the entire input surrounding the current target token  Models use the standard settings for the Big Transformer BIBREF16   The bi-directional model contains 353M parameters and the uni-directional model 190M parameters  Both models were trained for 1M steps using Nesterov's accelerated gradient BIBREF18 with momentum INLINEFORM3 following BIBREF19 
Context 3: Introduction
Rendering natural language descriptions from structured data is required in a wide variety of commercial applications such as generating descriptions of products, hotels, furniture, etc , from a corresponding table of facts about the entity  Such a table typically contains {field, value} pairs where the field is a property of the entity (e g , color) and the value is a set of possible assignments to this property (e g , color = red)  Another example of this is the recently introduced task of generating one line biography descriptions from a given Wikipedia infobox BIBREF0   The Wikipedia infobox serves as a table of facts about a person and the first sentence from the corresponding article serves as a one line description of the person  Figure FIGREF2 illustrates an example input infobox which contains fields such as Born, Residence, Nationality, Fields, Institutions and Alma Mater  Each field further contains some words (e g , particle physics, many-body theory, etc )  The corresponding description is coherent with the information contained in the infobox 
Note that the number of fields in the infobox and the ordering of the fields within the infobox varies from person to person  Given the large size (700K examples) and heterogeneous nature of the dataset which contains biographies of people from different backgrounds (sports, politics, arts, etc ), it is hard to come up with simple rule-based templates for generating natural language descriptions from infoboxes, thereby making a case for data-driven models  Based on the recent success of data-driven neural models for various other NLG tasks BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , one simple choice is to treat the infobox as a sequence of {field, value} pairs and use a standard seq2seq model for this task  However, such a model is too generic and does not exploit the specific characteristics of this task as explained below  First, note that while generating such descriptions from structured data, a human keeps track of information at two levels  Specifically, at a macro level, she would first decide which field to mention next and then at a micro level decide which of the values in the field needs to be mentioned next  For example, she first decides that at the current step, the field occupation needs attention and then decides which is the next appropriate occupation to attend to from the set of occupations (actor, director, producer, etc )  To enable this, we use a bifocal attention mechanism which computes an attention over fields at a macro level and over values at a micro level  We then fuse these attention weights such that the attention weight for a field also influences the attention over the values within it  Finally, we feed a fused context vector to the decoder which contains both field level and word level information  Note that such two-level attention mechanisms BIBREF6 , BIBREF7 , BIBREF8 have been used in the context of unstructured data (as opposed to structured data in our case), where at a macro level one needs to pay attention to sentences and at a micro level to words in the sentences 
Next, we observe that while rendering the output, once the model pays attention to a field (say, occupation) it needs to stay on this field for a few timesteps (till all the occupations are produced in the output)  We refer to this as the stay on behavior  Further, we note that once the tokens of a field are referred to, they are usually not referred to later  For example, once all the occupations have been listed in the output we will never visit the occupation field again because there is nothing left to say about it  We refer to this as the never look back behavior  To model the stay on behaviour, we introduce a forget (or remember) gate which acts as a signal to decide when to forget the current field (or equivalently to decide till when to remember the current field)  To model the never look back behaviour we introduce a gated orthogonalization mechanism which ensures that once a field is forgotten, subsequent field context vectors fed to the decoder are orthogonal to (or different from) the previous field context vectors 
We experiment with the WikiBio dataset BIBREF0 which contains around 700K {infobox, description} pairs and has a vocabulary of around 400K words  We show that the proposed model gives a relative improvement of 21% and 20% as compared to current state of the art models BIBREF0 , BIBREF9 on this dataset  The proposed model also gives a relative improvement of 10% as compared to the basic seq2seq model  Further, we introduce new datasets for French and German on the same lines as the English WikiBio dataset  Even on these two datasets, our model outperforms the state of the art methods mentioned above 
Related work
Natural Language Generation has always been of interest to the research community and has received a lot of attention in the past  The approaches for NLG range from (i) rule based approaches (e g , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 ) (ii) modular statistical approaches which divide the process into three phases (planning, selection and surface realization) and use data driven approaches for one or more of these phases BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 (iii) hybrid approaches which rely on a combination of handcrafted rules and corpus statistics BIBREF20 , BIBREF21 , BIBREF22 and (iv) the more recent neural network based models BIBREF1  
Neural models for NLG have been proposed in the context of various tasks such as machine translation BIBREF1 , document summarization BIBREF2 , BIBREF4 , paraphrase generation BIBREF23 , image captioning BIBREF24 , video summarization BIBREF25 , query based document summarization BIBREF5 and so on  Most of these models are data hungry and are trained on large amounts of data  On the other hand, NLG from structured data has largely been studied in the context of small datasets such as WeatherGov BIBREF26 , RoboCup BIBREF27 , NFL Recaps BIBREF15 , Prodigy-Meteo BIBREF28 and TUNA Challenge BIBREF29   Recently weather16 proposed RNN/LSTM based neural encoder-decoder models with attention for WeatherGov and RoboCup datasets 
Unlike the datasets mentioned above, the biography dataset introduced by lebret2016neural is larger (700K {table, descriptions} pairs) and has a much larger vocabulary (400K words as opposed to around 350 or fewer words in the above datasets)  Further, unlike the feed-forward neural network based model proposed by BIBREF0 we use a sequence to sequence model and introduce components to address the peculiar characteristics of the task  Specifically, we introduce neural components to address the need for attention at two levels and to address the stay on and never look back behaviour required by the decoder  KiddonZC16 have explored the use of checklists to track previously visited ingredients while generating recipes from ingredients  Note that two-level attention mechanisms have also been used in the context of summarization BIBREF6 , document classification BIBREF7 , dialog systems BIBREF8 , etc  However, these works deal with unstructured data (sentences at the higher level and words at a lower level) as opposed to structured data in our case 
Proposed model
As input we are given an infobox INLINEFORM0 , which is a set of pairs INLINEFORM1 where INLINEFORM2 corresponds to field names and INLINEFORM3 is the sequence of corresponding values and INLINEFORM4 is the total number of fields in INLINEFORM5   For example, INLINEFORM6 could be one such pair in this set  Given such an input, the task is to generate a description INLINEFORM7 containing INLINEFORM8 words  A simple solution is to treat the infobox as a sequence of fields followed by the values corresponding to the field in the order of their appearance in the infobox  For example, the infobox could be flattened to produce the following input sequence (the words in bold are field names which act as delimiters)
[Name] John Doe [Birth_Date] 19 March 1981 [Nationality] Indian      
The problem can then be cast as a seq2seq generation problem and can be modeled using a standard neural architecture comprising of three components (i) an input encoder (using GRU/LSTM cells), (ii) an attention mechanism to attend to important values in the input sequence at each time step and (iii) a decoder to decode the output one word at a time (again, using GRU/LSTM cells)  However, this standard model is too generic and does not exploit the specific characteristics of this task  We propose additional components, viz , (i) a fused bifocal attention mechanism which operates on fields (macro) and values (micro) and (ii) a gated orthogonalization mechanism to model stay on and never look back behavior 
Fused Bifocal Attention Mechanism
Intuitively, when a human writes a description from a table she keeps track of information at two levels  At the macro level, it is important to decide which is the appropriate field to attend to next and at a micro level (i e , within a field) it is important to know which values to attend to next  To capture this behavior, we use a bifocal attention mechanism as described below 
Macro Attention: Consider the INLINEFORM0 -th field INLINEFORM1 which has values INLINEFORM2   Let INLINEFORM3 be the representation of this field in the infobox  This representation can either be (i) the word embedding of the field name or (ii) some function INLINEFORM4 of the values in the field or (iii) a concatenation of (i) and (ii)  The function INLINEFORM5 could simply be the sum or average of the embeddings of the values in the field  Alternately, this function could be a GRU (or LSTM) which treats these values within a field as a sequence and computes the field representation as the final representation of this sequence (i e , the representation of the last time-step)  We found that bidirectional GRU is a better choice for INLINEFORM6 and concatenating the embedding of the field name with this GRU representation works best  Further, using a bidirectional GRU cell to take contextual information from neighboring fields also helps (these are the orange colored cells in the top-left block in Figure FIGREF3 with macro attention)  Given these representations INLINEFORM7 for all the INLINEFORM8 fields we compute an attention over the fields (macro level)  DISPLAYFORM0
where INLINEFORM0 is the state of the decoder at time step INLINEFORM1   INLINEFORM2 and INLINEFORM3 are parameters, INLINEFORM4 is the total number of fields in the input, INLINEFORM5 is the macro (field level) context vector at the INLINEFORM6 -th time step of the decoder 
Micro Attention: Let INLINEFORM0 be the representation of the INLINEFORM1 -th value in a given field  This representation could again either be (i) simply the embedding of this value (ii) or a contextual representation computed using a function INLINEFORM2 which also considers the other values in the field  For example, if INLINEFORM3 are the values in a field then these values can be treated as a sequence and the representation of the INLINEFORM4 -th value can be computed using a bidirectional GRU over this sequence  Once again, we found that using a bi-GRU works better then simply using the embedding of the value  Once we have such a representation computed for all values across all the fields, we compute the attention over these values (micro level) as shown below : DISPLAYFORM0
where INLINEFORM0 is the state of the decoder at time step INLINEFORM1   INLINEFORM2 and INLINEFORM3 are parameters, INLINEFORM4 is the total number of values across all the fields 
Fused Attention: Intuitively, the attention weights assigned to a field should have an influence on all the values belonging to the particular field  To ensure this, we reweigh the micro level attention weights based on the corresponding macro level attention weights  In other words, we fuse the attention weights at the two levels as: DISPLAYFORM0
where INLINEFORM0 is the field corresponding to the INLINEFORM1 -th value, INLINEFORM2 is the macro level context vector 
Gated Orthogonalization for Modeling Stay-On and Never Look Back behaviour
We now describe a series of choices made to model stay-on and never look back behavior  We first begin with the stay-on property which essentially implies that if we have paid attention to the field INLINEFORM0 at timestep INLINEFORM1 then we are likely to pay attention to the same field for a few more time steps  For example, if we are focusing on the occupation field at this timestep then we are likely to focus on it for the next few timesteps till all relevant values in this field have been included in the generated description  In other words, we want to remember the field context vector INLINEFORM2 for a few timesteps  One way of ensuring this is to use a remember (or forget) gate as given below which remembers the previous context vector when required and forgets it when it is time to move on from that field  DISPLAYFORM0
where INLINEFORM0 are parameters to be learned  The job of the forget gate is to ensure that INLINEFORM1 is similar to INLINEFORM2 when required (i e , by learning INLINEFORM3 when we want to continue focusing on the same field) and different when it is time to move on (by learning that INLINEFORM4 ) 
Next, the never look back property implies that once we have moved away from a field we are unlikely to pay attention to it again  For example, once we have rendered all the occupations in the generated description there is no need to return back to the occupation field  In other words, once we have moved on ( INLINEFORM0 ), we want the successive field context vectors INLINEFORM1 to be very different from the previous field vectors INLINEFORM2   One way of ensuring this is to orthogonalize successive field vectors using DISPLAYFORM0
where INLINEFORM0 is the dot product between vectors INLINEFORM1 and INLINEFORM2   The above equation essentially subtracts the component of INLINEFORM3 along INLINEFORM4   INLINEFORM5 is a learned parameter which controls the degree of orthogonalization thereby allowing a soft orthogonalization (i e , the entire component along INLINEFORM6 is not subtracted but only a fraction of it)  The above equation only ensures that INLINEFORM7 is soft-orthogonal to INLINEFORM8   Alternately, we could pass the sequence of context vectors, INLINEFORM9 generated so far through a GRU cell  The state of this GRU cell at each time step would thus be aware of the history of the field vectors till that timestep

ground_truth: No

No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What language model architectures are used?
Context 2:  The learning rate is linearly warmed up from INLINEFORM4 to 1 for 16K steps and then annealed using a cosine learning rate schedule with a single phase to 0 0001 BIBREF20   We train on 32 Nvidia V100 SXM2 GPUs and use the NCCL2 library as well as the torch distributed package for inter-GPU communication  Training relies on 16-bit floating point operations BIBREF21 and it took six days for the bi-directional model and four days for the uni-directional model 
Sequence to sequence model
We use the transformer implementation of the fairseq toolkit BIBREF22   The WMT en-de and en-tr experiments are based on the Big Transformer sequence to sequence architecture with 6 blocks in the encoder and decoder  For abstractive summarization we use a base transformer model BIBREF16   We tune dropout values of between 0 1 and 0 4 on the validation set  Models are optimized with Adam BIBREF23 using INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 and we use the same learning rate schedule as BIBREF16 ; we perform 10K-200K depending on bitext size  All models use label smoothing with a uniform prior distribution over the vocabulary INLINEFORM3 BIBREF24 , BIBREF25   We run experiments on 8 GPUs and generate translations with a beam of size 5 
Machine translation
We first present a comparison of the various strategies in different simulated parallel corpus size settings  For each experiment, we tune the dropout applied to the language model representations, and we reduce the number of optimizer steps for smaller bitext setups as models converge faster; all other hyper-parameters are equal between setups  Our baseline is a Big Transformer model and we also consider a variant where we share token embeddings between the encoder and decoder (; Inan et al , 2016; Press & Wolf, 2016) 
Figure FIGREF10 shows results averaged over six test sets relative to the baseline which does not share source and target embeddings (Appendix SECREF6 shows a detailed breakdown)  performs very well with little labeled data but the gains erode to practically zero in large bitext settings  Pre-trained language model representations are most effective in low bitext setups  The best performing strategy is ELMo embeddings input to the encoder ()  This improves the baseline by 3 8 BLEU in the 160K bitext setting and it still improves the 5 2M setting by over 1 BLEU 
We further improve by sharing learned word representations in the decoder by tying input and output embeddings ()  This configuration performs even better than with a gain of 5 3 BLEU in the 160K setup  Sharing decoder embeddings is equally applicable to   Language model representations are much less effective in the decoder: improves the 160K bitext setup but yields no improvements thereafter and performs even worse  We conjecture that pre-trained representations give much easier wins in the encoder  Table TABREF14 shows additional results on newstest2018 
Pre-trained representations mostly impacts the training time of the sequence to sequence model (see Appendix SECREF7 ): slows throughput during training by about 5 3x and is even slower because of the need to backpropagate through the LM for fine-tuning (9 2x)  However, inference is only 12-14% slower than the baseline when adding pre-trained embeddings to the encoder (, )  This is because the LM computation can be paralelized for all input tokens  Inference is much slower when adding representations to the decoder because the LM needs to be invoked repeatedly  Our current implementation does not cache LM operations for the previous state and can be made much faster 
The baseline uses a BPE vocabulary estimated on the language model corpora (§ SECREF3 )  Appendix SECREF6 shows that this vocabulary actually leads to sligtly better performance than a joint BPE code learned on the bitext as is usual 
Next, we validate our findings on the WMT'18 English-Turkish task for which the bitext is truly limited (208K sentence-pairs)  We use the language model vocab for the the English side of the bitext and a BPE vocabulary learned on the Turkish side  Table TABREF15 shows that ELMo embeddings for the encoder improve English-Turkish translation 
Abstractive summarization
Following BIBREF11 , we experiment on the non-anonymized version of   When generating summaries, we follow standard practice of tuning the maximum output length and disallow repeating the same trigram BIBREF27 , BIBREF14   For this task we train language model representations on the combination of newscrawl and the training data  Table TABREF16 shows that pre-trained embeddings can significantly improve on top of a strong baseline transformer  We also compare to BIBREF26 who use a task-specific architecture compared to our generic sequence to sequence baseline  Pre-trained representations are complementary to their method 
Conclusion
We presented an analysis of different strategies to add pre-trained language model representations to sequence to sequence models for neural machine translation and abstractive document summarization  Adding pre-trained representations is very effective for the encoder network and while returns diminish when more labeled data becomes available, we still observe improvements when millions of examples are available  In future research we will investigate ways to improve the decoder with pre-trained representations 
Context 3:  Now instead of orthogonalizing INLINEFORM10 to INLINEFORM11 we could orthogonalize INLINEFORM12 to the hidden state of this GRU at time-step INLINEFORM13   In practice, we found this to work better as it accounts for all the field vectors in the history instead of only the previous field vector 
In summary, Equation provides a mechanism for remembering the current field vector when appropriate (thus capturing stay-on behavior) using a remember gate  On the other hand, Equation EQREF10 explicitly ensures that the field vector is very different (soft-orthogonal) from the previous field vectors once it is time to move on (thus capturing never look back behavior)  The value of INLINEFORM0 computed in Equation EQREF10 is then used in Equation   The INLINEFORM1 (macro) thus obtained is then concatenated with INLINEFORM2 (micro) and fed to the decoder (see Fig  FIGREF3 )
Experimental setup
We now describe our experimental setup:
Datasets
We use the WikiBio dataset introduced by lebret2016neural  It consists of INLINEFORM0 biography articles from English Wikipedia  A biography article corresponds to a person (sportsman, politician, historical figure, actor, etc )  Each Wikipedia article has an accompanying infobox which serves as the structured input and the task is to generate the first sentence of the article (which typically is a one-line description of the person)  We used the same train, valid and test sets which were made publicly available by lebret2016neural 
We also introduce two new biography datasets, one in French and one in German  These datasets were created and pre-processed using the same procedure as outlined in lebret2016neural  Specifically, we extracted the infoboxes and the first sentence from the corresponding Wikipedia article  As with the English dataset, we split the French and German datasets randomly into train (80%), test (10%) and valid (10%)  The French and German datasets extracted by us has been made publicly available  The number of examples was 170K and 50K and the vocabulary size was 297K and 143K for French and German respectively  Although in this work we focus only on generating descriptions in one language, we hope that this dataset will also be useful for developing models which jointly learn to generate descriptions from structured data in multiple languages 
Models compared
We compare with the following models:
1  BIBREF0 : This is a conditional language model which uses a feed-forward neural network to predict the next word in the description conditioned on local characteristics (i e , words within a field) and global characteristics (i e , overall structure of the infobox) 
2  BIBREF9 : This model was proposed in the context of the WeatherGov and RoboCup datasets which have a much smaller vocabulary  They use an improved attention model with additional regularizer terms which influence the weights assigned to the fields 
3  Basic Seq2Seq: This is the vanilla encode-attend-decode model BIBREF1   Further, to deal with the large vocabulary ( INLINEFORM0 400K words) we use a copying mechanism as a post-processing step  Specifically, we identify the time steps at which the decoder produces unknown words (denoted by the special symbol UNK)  For each such time step, we look at the attention weights on the input words and replace the UNK word by that input word which has received maximum attention at this timestep  This process is similar to the one described in BIBREF30   Even lebret2016neural have a copying mechanism tightly integrated with their model 
Hyperparameter tuning
We tuned the hyperparameters of all the models using a validation set  As mentioned earlier, we used a bidirectional GRU cell as the function INLINEFORM0 for computing the representation of the fields and the values (see Section SECREF4 )  For all the models, we experimented with GRU state sizes of 128, 256 and 512  The total number of unique words in the corpus is around 400K (this includes the words in the infobox and the descriptions)  Of these, we retained only the top 20K words in our vocabulary (same as BIBREF0 )  We initialized the embeddings of these words with 300 dimensional Glove embeddings BIBREF31   We used Adam BIBREF32 with a learning rate of INLINEFORM1 , INLINEFORM2 and INLINEFORM3   We trained the model for a maximum of 20 epochs and used early stopping with the patience set to 5 epochs 
Results and Discussions
We now discuss the results of our experiments 
Comparison of different models
Following lebret2016neural, we used BLEU-4, NIST-4 and ROUGE-4 as the evaluation metrics  We first make a few observations based on the results on the English dataset (Table TABREF15 )  The basic seq2seq model, as well as the model proposed by weather16, perform better than the model proposed by lebret2016neural  Our final model with bifocal attention and gated orthogonalization gives the best performance and does 10% (relative) better than the closest baseline (basic seq2seq) and 21% (relative) better than the current state of the art method BIBREF0   In Table TABREF16 , we show some qualitative examples of the output generated by different models 
Human Evaluations
To make a qualitative assessment of the generated sentences, we conducted a human study on a sample of 500 Infoboxes which were sampled from English dataset  The annotators for this task were undergraduate and graduate students  For each of these infoboxes, we generated summaries using the basic seq2seq model and our final model with bifocal attention and gated orthogonalization  For each description and for each model, we asked three annotators to rank the output of the systems based on i) adequacy (i e  does it capture relevant information from the infobox), (ii) fluency (i e  grammar) and (iii) relative preference (i e , which of the two outputs would be preferred)  Overall the average fluency/adequacy (on a scale of 5) for basic seq2seq model was INLINEFORM0 and INLINEFORM1 for our model respectively 
The results from Table TABREF17 suggest that in general gated orthogonalization model performs better than the basic seq2seq model  Additionally, annotators were asked to verify if the generated summaries look natural (i e, as if they were generated by humans)  In 423 out of 500 cases, the annotators said “Yes” suggesting that gated orthogonalization model indeed produces good descriptions 
Performance on different languages
The results on the French and German datasets are summarized in Tables TABREF20 and TABREF20 respectively  Note that the code of BIBREF0 is not publicly available, hence we could not report numbers for French and German using their model  We observe that our final model gives the best performance - though the bifocal attention model performs poorly as compared to the basic seq2seq model on French  However, the overall performance for French and German are much smaller than those for English  There could be multiple reasons for this  First, the amount of training data in these two languages is smaller than that in English  Specifically, the amount of training data available in French (German) is only INLINEFORM0 ( INLINEFORM1 )% of that available for English  Second, on average the descriptions in French and German are longer than that in English (EN: INLINEFORM2 words, FR: INLINEFORM3 words and DE: INLINEFORM4 words)  Finally, a manual inspection across the three languages suggests that the English descriptions have a more consistent structure than the French descriptions  For example, most English descriptions start with name followed by date of birth but this is not the case in French  However, this is only a qualitative observation and it is hard to quantify this characteristic of the French and German datasets 
Visualizing Attention Weights
If the proposed model indeed works well then we should see attention weights that are consistent with the stay on and never look back behavior  To verify this, we plotted the attention weights in cases where the model with gated orthogonalization does better than the model with only bifocal attention  Figure FIGREF21 shows the attention weights corresponding to infobox in Figure FIGREF25   Notice that the model without gated orthogonalization has attention on both name field and article title while rendering the name  The model with gated orthogonalization, on the other hand, stays on the name field for as long as it is required but then moves and never returns to it (as expected) 
Due to lack of space, we do not show similar plots for French and German but we would like to mention that, in general, the differences between the attention weights learned by the model with and without gated orthogonalization were more pronounced for the French/German dataset than the English dataset  This is in agreement with the results reported in Table TABREF20 and TABREF20 where the improvements given by gated orthogonalization are more for French/German than for English 
Out of domain results
What if the model sees a different INLINEFORM0 of person at test time? For example, what if the training data does not contain any sportspersons but at test time we encounter the infobox of a sportsperson  This is the same as seeing out-of-domain data at test time  Such a situation is quite expected in the products domain where new products with new features (fields) get frequently added to the catalog  We were interested in three questions here  First, we wanted to see if testing the model on out-of-domain data indeed leads to a drop in the performance  For this, we compared the performance of our best model in two scenarios (i) trained on data from all domains (including the target domain) and tested on the target domain (sports, arts) and (ii) trained on data from all domains except the target domain and tested on the target domain  Comparing rows 1 and 2 of Table TABREF32 we observed a significant drop in the performance  Note that the numbers for sports domain in row 1 are much better than the Arts domain because roughly 40% of the WikiBio training data contains sportspersons 
Next, we wanted to see if we can use a small amount of data from the target domain to fine tune a model trained on the out of domain data  We observe that even with very small amounts of target domain data the performance starts improving significantly (see rows 3 and 4 of Table TABREF32 )  Note that if we train a model from scratch with only limited data from the target domain instead of fine-tuning a model trained on a different source domain then the performance is very poor  In particular, training a model from scratch with 10K training instances we get a BLEU score of INLINEFORM0 and INLINEFORM1 for arts and sports respectively  Finally, even though the actual words used for describing a sportsperson (footballer, cricketer, etc ) would be very different from the words used to describe an artist (actor, musician, etc ) they might share many fields (for example, date of birth, occupation, etc )  As seen in Figure FIGREF28 (attention weights corresponding to the infobox in Figure FIGREF27 ), the model predicts the attention weights correctly for common fields (such as occupation) but it is unable to use the right vocabulary to describe the occupation (since it has not seen such words frequently in the training data)  However, once we fine tune the model with limited data from the target domain we see that it picks up the new vocabulary and produces a correct description of the occupation 
Conclusion
We present a model for generating natural language descriptions from structured data  To address specific characteristics of the problem we propose neural components for fused bifocal attention and gated orthogonalization to address stay on and never look back behavior while decoding  Our final model outperforms an existing state of the art model on a large scale WikiBio dataset by 21%  We also introduce datasets for French and German and demonstrate that our model gives state of the art results on these datasets  Finally, we perform experiments with an out-of-domain model and show that if such a model is fine-tuned with small amounts of in domain data then it can give an improved performance on the target domain 
Given the multilingual nature of the new datasets, as future work, we would like to build models which can jointly learn to generate natural language descriptions from structured data in multiple languages  One idea is to replace the concepts in the input infobox by Wikidata concept ids which are language agnostic  A large amount of input vocabulary could thus be shared across languages thereby facilitating joint learning 
Acknowledgements
We thank Google for supporting Preksha Nema through their Google India Ph D  Fellowship program  We also thank Microsoft Research India for supporting Shreyas Shetty through their generous travel grant for attending the conference 

ground_truth: uni-directional model to augment the decoder

bi-directional language model to augment the sequence to sequence encoder ,  uni-directional model to augment the decoder

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they use pretrained embeddings?
Context 4: g , “No dizziness, oh wait    last week I felt a bit dizzy when biking” 
Response with Topic Drift: When a symptom/topic like headache is inquired, the response might be: “Only some chest pain at night”, not referring to the original symptom (headache) at all 
Response with Transitional Clauses: Repeating some of the previous content, but often unrelated to critical clinical information and usually followed by topic drift  For example, “Swelling    swelling    I don't cough at night” 
Simulating Symptom Monitoring Dataset for Training
We divide the construction of data simulation into two stages  In Section SECREF16 , we build templates and expression pools using linguistic analysis followed by manual verification  In Section SECREF20 , we present our proposed framework for generating simulated training data  The templates and framework are verified for logical correctness and clinical soundness 
Template Construction
Each utterance in the seed data is categorized according to Table TABREF14 and then abstracted into templates by replacing entity phrases like cough and often with respective placeholders “#symptom#” and “#frequency#”  The templates are refined through verifying logical correctness and injecting expression diversity by linguistically trained researchers  As these replacements do not alter the syntactic structure, we interchange such placeholders with various verbal expressions to enlarge the simulated training set in Section SECREF20   Clinical validation was also conducted by certified telehealth nurses 
For the 9 symptoms (e g  chest pain, cough) and 5 attributes (e g , extent, frequency), we collect various expressions from the seed data, and expand them through synonym replacement  Some attributes are unique to a particular symptom; e g , “left leg” in #location# is only suitable to describe the symptom swelling, but not the symptom headache  Therefore, we only reuse general expressions like “slight” in #extent# across different symptoms to diversify linguistic expressions 
Two linguistically trained researchers constructed expression pools for each symptom and each attribute to account for different types of paraphrasing and descriptions  These expression pools are used in Section SECREF20 (c) 
Simulated Data Generation Framework
Figure FIGREF15 shows the five steps we use to generate multi-turn symptom monitoring dialogue samples 
(a) Topic Selection: While nurses might prefer to inquire the symptoms in different orders depending on the patient's history, our preliminary analysis shows that modeling results do not differ noticeably if topics are of equal prior probabilities  Thus we adopt this assumption for simplicity 
(b) Template Selection: For each selected topic, one inquiry template and one response template are randomly chosen to compose a turn  To minimize adverse effects of underfitting, we redistributed the frequency distribution in Table TABREF14 : For utterance types that are below 15%, we boosted them to 15%, and the overall relative distribution ranking is balanced and consistent with Table TABREF14  
(c) Enriching Linguistic Expressions: The placeholders in the selected templates are substituted with diverse expressions from the expression pools in Section UID19 to characterize the symptoms and their corresponding attributes 
(d) Multi-Turn Dialogue State Tracking: A greedy algorithm is applied to complete conversations  A “completed symptoms” list and a “to-do symptoms” list are used for symptom topic tracking  We also track the “completed attributes" and “to-do attributes"  For each symptom, all related attributes are iterated  A dialogue ends only when all possible entities are exhausted, generating a multi-turn dialogue sample, which encourages the model to learn from the entire discussion flow rather than a single turn to comprehend contextual dependency  The average length of a simulated dialogue is 184 words, which happens to be twice as long as an average dialogue from the real-world evaluation set  Moreover, to model the roles of the respondents, we set the ratio between patients and caregivers to 2:1; this statistic is inspired by the real scenarios in the seed dataset  For both the caregivers and patients, we assume equal probability of both genders  The corresponding pronouns in the conversations are thus determined by the role and gender of these settings 
(e) Multi-Turn Sample Annotation: For each multi-turn dialogue, a query is specified by a symptom and an attribute  The groundtruth output of the QA system is automatically labeled based on the template generation rules, but also manually verified to ensure annotation quality  Moreover, we adopt the unanswerable design in BIBREF6 : when the patient does not mention a particular symptom, the answer is defined as “No Answer”  This process is repeated until all logical permutations of symptoms and attributes are exhausted 
Experiments
Model Design
We implemented an established model in reading comprehension, a bi-directional attention pointer network BIBREF1 , and equipped it with an answerable classifier, as depicted in Figure FIGREF21   First, tokens in the given dialogue INLINEFORM0 and query INLINEFORM1 are converted into embedding vectors  Then the dialogue embeddings are fed to a bi-directional LSTM encoding layer, generating a sequence of contextual hidden states  Next, the hidden states and query embeddings are processed by a bi-directional attention layer, fusing attention information in both context-to-query and query-to-context directions  The following two bi-directional LSTM modeling layers read the contextual sequence with attention  Finally, two respective linear layers with softmax functions are used to estimate token INLINEFORM2 's INLINEFORM3 and INLINEFORM4 probability of the answer span INLINEFORM5  
In addition, we add a special tag “[SEQ]” at the head of INLINEFORM0 to account for the case of “No answer” BIBREF4 and adopt an answerable classifier as in BIBREF25   More specifically, when the queried symptom or attribute is not mentioned in the dialogue, the answer span should point to the tag “[SEQ]” and answerable probability should be predicted as 0 
Implementation Details
The model was trained via gradient backpropagation with the cross-entropy loss function of answer span prediction and answerable classification, optimized by Adam algorithm BIBREF26 with initial learning rate of INLINEFORM0   Pre-trained GloVe BIBREF3 embeddings (size INLINEFORM1 ) were used  We re-shuffled training samples at each epoch (batch size INLINEFORM2 )  Out-of-vocabulary words ( INLINEFORM3 ) were replaced with a fixed random vector  L2 regularization and dropout (rate INLINEFORM4 ) were used to alleviate overfitting BIBREF27  
Evaluation Setup
To evaluate the effectiveness of our linguistically-inspired simulation approach, the model is trained on the simulated data (see Section SECREF20 )  We designed 3 evaluation sets: (1) Base Set (1,264 samples) held out from the simulated data  (2) Augmented Set (1,280 samples) built by adding two out-of-distribution symptoms, with corresponding dialogue contents and queries, to the Base Set (“bleeding” and “cold”, which never appeared in training data)  (3) Real-World Set (944 samples) manually delineated from the the symptom checking portions (approximately 4 hours) of real-world dialogues, and annotated as evaluation samples 
Results
Evaluation results are in Table TABREF25 with exact match (EM) and F1 score in BIBREF0 metrics  To distinguish the correct answer span from the plausible ones which contain the same words, we measure the scores on the position indices of tokens  Our results show that both EM and F1 score increase with training sample size growing and the optimal size in our setting is 100k  The best-trained model performs well on both the Base Set and the Augmented Set, indicating that out-of-distribution symptoms do not affect the comprehension of existing symptoms and outputs reasonable answers for both in- and out-of-distribution symptoms  On the Real-World Set, we obtained 78 23 EM score and 80 18 F1 score respectively 
Error analysis suggests the performance drop from the simulated test sets is due to the following: 1) sparsity issues resulting from the expression pools excluding various valid but sporadic expressions  2) nurses and patients occasionally chit-chat in the Real-World Set, which is not simulated in the training set  At times, these chit-chats make the conversations overly lengthy, causing the information density to be lower  These issues could potentially distract and confuse the comprehension model  3) an interesting type of infrequent error source, caused by patients elaborating on possible causal relations of two symptoms  For example, a patient might say “My giddiness may be due to all this cough”  We are currently investigating how to close this performance gap efficiently 
Ablation Analysis
To assess the effectiveness of bi-directional attention, we bypassed the bi-attention layer by directly feeding the contextual hidden states and query embeddings to the modeling layer  To evaluate the pre-trained GloVe embeddings, we randomly initialized and trained the embeddings from scratch  These two procedures lead to 10% and 18% performance degradation on the Augmented Set and Real-World Set, respectively (see Table TABREF27 ) 
Conclusion
We formulated a dialogue comprehension task motivated by the need in telehealth settings to extract key clinical information from spoken conversations between nurses and patients  We analyzed linguistic characteristics of real-world human-human symptom checking dialogues, constructed a simulated dataset based on linguistically inspired and clinically validated templates, and prototyped a QA system  The model works effectively on a simulated test set using symptoms excluded during training and on real-world conversations between nurses and patients  We are currently improving the model's dialogue comprehension capability in complex reasoning and context understanding and also applying the QA model to summarization and virtual nurse applications 
Acknowledgements
Research efforts were supported by funding for Digital Health and Deep Learning I2R (DL2 SSF Project No: A1718g0045) and the Science and Engineering Research Council (SERC Project No: A1818g0044), A*STAR, Singapore  In addition, this work was conducted using resources and infrastructure provided by the Human Language Technology unit at I2R  The telehealth data acquisition was funded by the Economic Development Board (EDB), Singapore Living Lab Fund and Philips Electronics – Hospital to Home Pilot Project (EDB grant reference number: S14-1035-RF-LLF H and W) 
We acknowledge valuable support and assistance from Yulong Lin, Yuanxin Xiang, and Ai Ti Aw at the Institute for Infocomm Research (I2R); Weiliang Huang at the Changi General Hospital (CGH) Department of Cardiology, Hong Choon Oh at CGH Health Services Research, and Edris Atikah Bte Ahmad, Chiu Yan Ang, and Mei Foon Yap of the CGH Integrated Care Department 
We also thank Eduard Hovy and Bonnie Webber for insightful discussions and the anonymous reviewers for their precious feedback to help improve and extend this piece of work 

ground_truth: Yes

Yes