prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What metrics are used to establish that this makes chatbots more knowledgeable and better at learning and conversation? 
Context 1: Introduction
Chatbots such as dialog and question-answering systems have a long history in AI and natural language processing  Early such systems were mostly built using markup languages such as AIML, handcrafted conversation generation rules, and/or information retrieval techniques BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   Recent neural conversation models BIBREF4 , BIBREF5 , BIBREF6 are even able to perform open-ended conversations  However, since they do not use explicit knowledge bases and do not perform inference, they often suffer from generic and dull responses BIBREF5 , BIBREF7   More recently, BIBREF8 and BIBREF9 proposed to use knowledge bases (KBs) to help generate responses for knowledge-grounded conversation  However, one major weakness of all existing chat systems is that they do not explicitly or implicitly learn new knowledge in the conversation process  This seriously limits the scope of their applications  In contrast, we humans constantly learn new knowledge in our conversations  Even if some existing systems can use very large knowledge bases either harvested from a large data source such as the Web or built manually, these KBs still miss a large number of facts (knowledge) BIBREF10   It is thus important for a chatbot to continuously learn new knowledge in the conversation process to expand its KB and to improve its conversation ability 
In recent years, researchers have studied the problem of KB completion, i e , inferring new facts (knowledge) automatically from existing facts in a KB  KB completion (KBC) is defined as a binary classification problem: Given a query triple, ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ), we want to predict whether the source entity INLINEFORM3 and target entity INLINEFORM4 can be linked by the relation INLINEFORM5   However, existing approaches BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 solve this problem under the closed-world assumption, i e , INLINEFORM6 , INLINEFORM7 and INLINEFORM8 are all known to exist in the KB  This is a major weakness because it means that no new knowledge or facts may contain unknown entities or relations  Due to this limitation, KBC is clearly not sufficient for knowledge learning in conversations because in a conversation, the user can say anything, which may contain entities and relations that are not already in the KB 
In this paper, we remove this assumption of KBC, and allow all INLINEFORM0 , INLINEFORM1 and INLINEFORM2 to be unknown  We call the new problem open-world knowledge base completion (OKBC)  OKBC generalizes KBC  Below, we show that solving OKBC naturally provides the ground for knowledge learning and inference in conversations  In essence, we formulate an abstract problem of knowledge learning and inference in conversations as a well-defined OKBC problem in the interactive setting 
From the perspective of knowledge learning in conversations, essentially we can extract two key types of information, true facts and queries, from the user utterances  Queries are facts whose truth values need to be determined  Note that we do not study fact or relation extraction in this paper as there is an extensive work on the topic  (1) For a true fact, we will incorporate it into the KB  Here we need to make sure that it is not already in the KB, which involves relation resolution and entity linking  After a fact is added to the KB, we may predict that some related facts involving some existing relations in the KB may also be true (not logical implications as they can be automatically inferred)  For example, if the user says “Obama was born in USA,” the system may guess that (Obama, CitizenOf, USA) (meaning that Obama is a citizen of USA) could also be true based on the current KB  To verify this fact, it needs to solve a KBC problem by treating (Obama, CitizenOf, USA) as a query  This is a KBC problem because the fact (Obama, BornIn, USA) extracted from the original sentence has been added to the KB  Then Obama and USA are in the KB  If the KBC problem is solved, it learns a new fact (Obama, CitizenOf, USA) in addition to the extracted fact (Obama, BornIn, USA)  (2) For a query fact, e g , (Obama, BornIn, USA) extracted from the user question “Was Obama born in USA?” we need to solve the OKBC problem if any of “Obama, “BornIn”, or “USA" is not already in the KB 
We can see that OKBC is the core of a knowledge learning engine for conversation  Thus, in this paper, we focus on solving it  We assume that other tasks such as fact/relation extraction and resolution and guessing of related facts of an extracted fact are solved by other sub-systems 
We solve the OKBC problem by mimicking how humans acquire knowledge and perform reasoning in an interactive conversation  Whenever we encounter an unknown concept or relation while answering a query, we perform inference using our existing knowledge  If our knowledge does not allow us to draw a conclusion, we typically ask questions to others to acquire related knowledge and use it in inference  The process typically involves an inference strategy (a sequence of actions), which interleaves a sequence of processing and interactive actions  A processing action can be the selection of related facts, deriving inference chain, etc , that advances the inference process  An interactive action can be deciding what to ask, formulating a suitable question, etc , that enable us to interact  The process helps grow the knowledge over time and the gained knowledge enables us to communicate better in the future  We call this lifelong interactive learning and inference (LiLi)  Lifelong learning is reflected by the facts that the newly acquired facts are retained in the KB and used in inference for future queries, and that the accumulated knowledge in addition to the updated KB including past inference performances are leveraged to guide future interaction and learning  LiLi should have the following capabilities:
This setting is ideal for many NLP applications like dialog and question-answering systems that naturally provide the scope for human interaction and demand real-time inference 
LiLi starts with the closed-world KBC approach path-ranking (PR) BIBREF11 , BIBREF17 and extends KBC in a major way to open-world knowledge base completion (OKBC)  For a relation INLINEFORM0 , PR works by enumerating paths (except single-link path INLINEFORM1 ) between entity-pairs linked by INLINEFORM2 in the KB and use them as features to train a binary classifier to predict whether a query INLINEFORM3 should be in the KB  Here, a path between two entities is a sequence of relations linking them  In our work, we adopt the latest PR method, C-PR BIBREF16 and extend it to make it work in the open-world setting  C-PR enumerates paths by performing bidirectional random walks over the KB graph while leveraging the context of the source-target entity-pair  We also adopt and extend the compositional vector space model BIBREF20 , BIBREF21 with continual learning capability for prediction 
Given an OKBC query ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ) (e g , “(Obama, CitizenOf, USA), which means whether Obama a citizen of USA), LiLi interacts with the user (if needed) by dynamically formulating questions (see the interaction example in Figure 1, which will be further explained in §3) and leverages the interactively acquired knowledge (supporting facts (SFs) in the figure) for continued inference  To do so, LiLi formulates a query-specific inference strategy and executes it  We design LiLi in a Reinforcement Learning (RL) setting that performs sub-tasks like formulating and executing strategy, training a prediction model for inference, and knowledge retention for future use  To the best of our knowledge, our work is the first to address the OKBC problem and to propose an interactive learning mechanism to solve it in a continuous or lifelong manner  We empirically verify the effectiveness of LiLi on two standard real-world KBs: Freebase and WordNet  Experimental results show that LiLi is highly effective in terms of its predictive performance and strategy formulation ability 
Related Work
To the best of our knowledge, we are not aware of any knowledge learning system that can learn new knowledge in the conversation process  This section thus discusses other related work 
Among existing KB completion approaches, BIBREF20 extended the vector space model for zero-shot KB inference  However, the model cannot handle unknown entities and can only work on fixed set of unknown relations with known embeddings  Recently, BIBREF22 proposed a method using external text corpus to perform inference on unknown entities  However, the method cannot handle unknown relations  Thus, these methods are not suitable for our open-world setting  None of the existing KB inference methods perform interactive knowledge learning like LiLi  NELL BIBREF23 continuously updates its KB using facts extracted from the Web  Our task is very different as we do not do Web fact extraction (which is also useful)  We focus on user interactions in this paper  Our work is related to interactive language learning (ILL) BIBREF24 , BIBREF25 , but these are not about KB completion  The work in BIBREF26 allows a learner to ask questions in dialogue  However, this work used RL to learn about whether to ask the user or not  The “what to ask aspect" was manually designed by modeling synthetic tasks  LiLi formulates query-specific inference strategies which embed interaction behaviors  Also, no existing dialogue systems BIBREF4 , BIBREF27 , BIBREF28 , BIBREF29 , BIBREF30 employ lifelong learning to train prediction models by using information/knowledge retained in the past 
Our work is related to general lifelong learning in BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36   However, they learn only one type of tasks, e g , supervised, topic modeling or reinforcement learning (RL) tasks  None of them is suitable for our setting, which involves interleaving of RL, supervised and interactive learning  More details about lifelong learning can be found in the book BIBREF31  
Interactive Knowledge Learning (LiLi)
We design LiLi as a combination of two interconnected models: (1) a RL model that learns to formulate a query-specific inference strategy for performing the OKBC task, and (2) a lifelong prediction model to predict whether a triple should be in the KB, which is invoked by an action while executing the inference strategy and is learned for each relation as in C-PR  The framework improves its performance over time through user interaction and knowledge retention  Compared to the existing KB inference methods, LiLi overcomes the following three challenges for OKBC:
1  Mapping open-world to close-world  Being a closed-world method, C-PR cannot extract path features and learn a prediction model when any of INLINEFORM0 , INLINEFORM1 or INLINEFORM2 is unknown  LiLi solves this problem through interactive knowledge acquisition  If INLINEFORM3 is unknown, LiLi asks the user to provide a clue (an example of INLINEFORM4 )  And if INLINEFORM5 or INLINEFORM6 is unknown, LiLi asks the user to provide a link (relation) to connect the unknown entity with an existing entity (automatically selected) in the KB  We refer to such a query as a connecting link query (CLQ)  The acquired knowledge reduces OKBC to KBC and makes the inference task feasible 
2  Spareseness of KB  A main issue of all PR methods like C-PR is the connectivity of the KB graph  If there is no path connecting INLINEFORM0 and INLINEFORM1 in the graph, path enumeration of C-PR gets stuck and inference becomes infeasible  In such cases, LiLi uses a template relation (“@-?-@") as the missing link marker to connect entity-pairs and continues feature extraction  A path containing “@-?-@" is called an incomplete path  Thus, the extracted feature set contains both complete (no missing link) and incomplete paths  Next, LiLi selects an incomplete path from the feature set and asks the user to provide a link for path completion  We refer to such a query as missing link query (MLQ) 
3  Limitation in user knowledge  If the user is unable to respond to MLQs or CLQs, LiLi uses a guessing mechanism (discussed later) to fill the gap  This enables LiLi to continue its inference even if the user cannot answer a system question 
Components of LiLi
As lifelong learning needs to retain knowledge learned from past tasks and use it to help future learning BIBREF31 , LiLi uses a Knowledge Store (KS) for knowledge retention  KS has four components: (i) Knowledge Graph ( INLINEFORM0 ): INLINEFORM1 (the KB) is initialized with base KB triples (see §4) and gets updated over time with the acquired knowledge  (ii) Relation-Entity Matrix ( INLINEFORM2 ): INLINEFORM3 is a sparse matrix, with rows as relations and columns as entity-pairs and is used by the prediction model  Given a triple ( INLINEFORM4 , INLINEFORM5 , INLINEFORM6 ) INLINEFORM7 , we set INLINEFORM8 [ INLINEFORM9 , ( INLINEFORM10 , INLINEFORM11 )] = 1 indicating INLINEFORM12 occurs for pair ( INLINEFORM13 , INLINEFORM14 )  (iii) Task Experience Store ( INLINEFORM15 ): INLINEFORM16 stores the predictive performance of LiLi on past learned tasks in terms of Matthews correlation coefficient (MCC) that measures the quality of binary classification  So, for two tasks INLINEFORM17 and INLINEFORM18 (each relation is a task), if INLINEFORM19 [ INLINEFORM20 ] INLINEFORM21 INLINEFORM22 [ INLINEFORM23 ] [where INLINEFORM24 [ INLINEFORM25 ]=MCC( INLINEFORM26 )], we say C-PR has learned INLINEFORM27 well compared to INLINEFORM28   (iv) Incomplete Feature DB ( INLINEFORM29 ): INLINEFORM30 stores the frequency of an incomplete path INLINEFORM31 in the form of a tuple ( INLINEFORM32 , INLINEFORM33 , INLINEFORM34 ) and is used in formulating MLQs  INLINEFORM35 [( INLINEFORM36 , INLINEFORM37 , INLINEFORM38 )] = INLINEFORM39 implies LiLi has extracted incomplete path INLINEFORM40 INLINEFORM41 times involving entity-pair INLINEFORM42 [( INLINEFORM43 , INLINEFORM44 )] for query relation INLINEFORM45  
The RL model learns even after training whenever it encounters an unseen state (in testing) and thus, gets updated over time  KS is updated continuously over time as a result of the execution of LiLi and takes part in future learning  The prediction model uses lifelong learning (LL), where we transfer knowledge (parameter values) from the model for a past most similar task to help learn for the current task  Similar tasks are identified by factorizing INLINEFORM0 and computing a task similarity matrix INLINEFORM1   Besides LL, LiLi uses INLINEFORM2 to identify poorly learned past tasks and acquire more clues for them to improve its skillset over time 
LiLi also uses a stack, called Inference Stack ( INLINEFORM0 ) to hold query and its state information for RL  LiLi always processes stack top ( INLINEFORM1 [top])  The clues from the user get stored in INLINEFORM2 on top of the query during strategy execution and processed first  Thus, the prediction model for INLINEFORM3 is learned before performing inference on query, transforming OKBC to a KBC problem  Table 1 shows the parameters of LiLi used in the following sections 
Working of LiLi
Given an OKBC query ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ), we represent it as a data instance INLINEFORM3   INLINEFORM4 consists of INLINEFORM5 (the query triple), INLINEFORM6 (interaction limit set for INLINEFORM7 ), INLINEFORM8 (experience list storing the transition history of MDP for INLINEFORM9 in RL) and INLINEFORM10 (mode of INLINEFORM11 ) denoting if INLINEFORM12 is ` INLINEFORM13 ' (training), ` INLINEFORM14 ' (validation), ` INLINEFORM15 ' (evaluation) or ` INLINEFORM16 ' (clue) instance and INLINEFORM17 (feature set)  We denote INLINEFORM18 ( INLINEFORM19 ) as the set of all complete (incomplete) path features in INLINEFORM20   Given a data instance INLINEFORM21 , LiLi starts its initialization as follows: it sets the state as INLINEFORM22 (based on INLINEFORM23 , explained later), pushes the query tuple ( INLINEFORM24 , INLINEFORM25 ) into INLINEFORM26 and feeds INLINEFORM27 [top] to the RL-model for strategy formulation from INLINEFORM28  
Inference Strategy Formulation  We view solving the strategy formulation problem as learning to play an inference game, where the goal is to formulate a strategy that "makes the inference task possible"  Considering PR methods, inference is possible, iff (1) INLINEFORM0 becomes known to its KB (by acquiring clues when INLINEFORM1 is unknown) and (2) path features are extracted between INLINEFORM2 and INLINEFORM3 (which inturn requires INLINEFORM4 and INLINEFORM5 to be known to KB)  If these conditions are met at the end of an episode (when strategy formulation finishes for a given query) of the game, LiLi wins and thus, it trains the prediction model for INLINEFORM6 and uses it for inference 
LiLi's strategy formulation is modeled as a Markov Decision Process (MDP) with finite state ( INLINEFORM0 ) and action ( INLINEFORM1 ) spaces  A state INLINEFORM2 consists of 10 binary state variables (Table 2), each of which keeps track of results of an action INLINEFORM3 taken by LiLi and thus, records the progress in inference process made so far  INLINEFORM4 is the initial state with all state bits set as 0  If the data instance (query) is a clue [ INLINEFORM5 ], INLINEFORM6 [CLUE] is set as 1  INLINEFORM7 consists of 6 actions (Table 3)  INLINEFORM8 , INLINEFORM9 , INLINEFORM10 are processing actions and INLINEFORM11 , INLINEFORM12 , INLINEFORM13 are interactive actions  Whenever INLINEFORM14 is executed, the MDP reaches the terminal state  Given an action INLINEFORM15 in state INLINEFORM16 , if INLINEFORM17 is invalid in INLINEFORM21 or the objective of INLINEFORM22 is unsatisfied (* marked the condition in INLINEFORM23 ), RL receives a negative reward (empirically set); else receives a positive reward   We use Q-learning BIBREF38 with INLINEFORM24 -greedy strategy to learn the optimal policy for training the RL model  Note that, the inference strategy is independent of KB type and correctness of prediction  Thus, the RL-model is trained only once from scratch (reused thereafter for other KBs) and also, independently of the prediction model 
Sometimes the training dataset may not be enough to learn optimal policy for all INLINEFORM0   Thus, encountering an unseen state during test can make RL-model clueless about the action  Given a state INLINEFORM1 , whenever an invalid INLINEFORM2 is chosen, LiLi remains in INLINEFORM3   For INLINEFORM4 , LiLi remains in INLINEFORM5 untill INLINEFORM6 (see Table 1 for INLINEFORM7 )  So, if the state remains the same for ( INLINEFORM8 +1) times, it implies LiLi has encountered a fault (an unseen state)  RL-model instantly switches to the training mode and randomly explores INLINEFORM9 to learn the optimal action (fault-tolerant learning)  While exploring INLINEFORM10 , the model chooses INLINEFORM11 only when it has tried all other INLINEFORM12 to avoid abrupt end of episode 
Execution of Actions  At any given point in time, let ( INLINEFORM0 , INLINEFORM1 ) be the current INLINEFORM2 [top], INLINEFORM3 is the chosen action and the current version of KS components are INLINEFORM4 , INLINEFORM5 , INLINEFORM6 and INLINEFORM7   Then, if INLINEFORM8 is invalid in INLINEFORM9 , LiLi only updates INLINEFORM10 [top] with ( INLINEFORM11 , INLINEFORM12 ) and returns INLINEFORM13 [top] to RL-model  In this process, LiLi adds experience ( INLINEFORM14 , INLINEFORM15 , INLINEFORM16 , INLINEFORM17 ) in INLINEFORM18 and then, replaces INLINEFORM19 [top] with ( INLINEFORM20 , INLINEFORM21 )  If INLINEFORM22 is valid in INLINEFORM23 , LiLi first sets the next state INLINEFORM24 and performs a sequence of operations INLINEFORM25 based on INLINEFORM26 (discussed below)  Unless specified, in INLINEFORM27 , LiLi always monitors INLINEFORM28 and if INLINEFORM29 becomes 0, LiLi sets INLINEFORM30   Also, whenever LiLi asks the user a query, INLINEFORM31 is decremented by 1  Once INLINEFORM32 ends, LiLi updates INLINEFORM33 [top] with ( INLINEFORM34 , INLINEFORM35 ) and returns INLINEFORM36 [top] to RL-model for choosing the next action 
In INLINEFORM0 , LiLi searches INLINEFORM1 , INLINEFORM2 , INLINEFORM3 in INLINEFORM4 and sets appropriate bits in INLINEFORM5 (see Table 2)  If INLINEFORM6 was unknown before and is just added to INLINEFORM7 or is in the bottom INLINEFORM8 % (see Table 1 for INLINEFORM9 ) of INLINEFORM10 , LiLi randomly sets INLINEFORM14 with probability INLINEFORM15   If INLINEFORM16 is a clue and INLINEFORM17 , LiLi updates KS with triple INLINEFORM18 , where ( INLINEFORM19 , INLINEFORM20 , INLINEFORM21 ) and ( INLINEFORM22 , INLINEFORM23 , INLINEFORM24 ) gets added to INLINEFORM25 and INLINEFORM26 , INLINEFORM27 are set as 1 
In INLINEFORM0 , LiLi asks the user to provide a clue (+ve instance) for INLINEFORM1 and corrupts INLINEFORM2 and INLINEFORM3 of the clue once at a time, to generate -ve instances by sampling nodes from INLINEFORM4   These instances help in training prediction model for INLINEFORM5 while executing INLINEFORM6  
In INLINEFORM0 , LiLi selects an incomplete path INLINEFORM1 from INLINEFORM2 to formulate MLQ, such that INLINEFORM3 is most frequently observed for INLINEFORM4 and INLINEFORM5 is high, given by INLINEFORM6   Here, INLINEFORM7 denotes the contextual similarity BIBREF16 of entity-pair INLINEFORM8   If INLINEFORM9 is high, INLINEFORM10 is more likely to possess a relation between them and so, is a good candidate for formulating MLQ  When the user does not respond to MLQ (or CLQ in INLINEFORM11 ), the guessing mechanism is used, which works as follows: Since contextual similarity of entity-pairs is highly correlated with their class labels BIBREF16 , LiLi divides the similarity range [-1, 1] into three segments, using a low ( INLINEFORM12 ) and high ( INLINEFORM13 ) similarity threshold and replaces the missing link with INLINEFORM14 in INLINEFORM15 to make it complete as follows: If INLINEFORM16 , INLINEFORM17 = “@-LooselyRelatedTo-@"; else if INLINEFORM18 , INLINEFORM19 =“@-NotRelatedTo-@"; Otherwise, INLINEFORM20 =“@-RelatedTo-@" 
In INLINEFORM0 , LiLi asks CLQs for connecting unknown entities INLINEFORM1 and/or INLINEFORM2 with INLINEFORM3 by selecting the most contextually relevant node (wrt INLINEFORM4 , INLINEFORM5 ) from INLINEFORM6 , given by link INLINEFORM7   We adopt the contextual relevance idea in BIBREF16 which is computed using word embedding BIBREF39
In INLINEFORM0 , LiLi extracts path features INLINEFORM1 between ( INLINEFORM2 , INLINEFORM3 ) and updates INLINEFORM4 with incomplete features from INLINEFORM5   LiLi always trains the prediction model with complete features INLINEFORM6 and once INLINEFORM7 or INLINEFORM8 , LiLi stops asking MLQs  Thus, in both INLINEFORM9 and INLINEFORM10 , LiLi always monitors INLINEFORM11 to check for the said requirements and sets INLINEFORM12 to control interactions 
In INLINEFORM0 , if LiLi wins the episode, it adds INLINEFORM1 in one of data buffers INLINEFORM2 based on its mode INLINEFORM3   E g , if INLINEFORM4 or INLINEFORM5 , INLINEFORM6 is used for training and added to INLINEFORM7   Similarly validation buffer INLINEFORM8 and evaluation buffer INLINEFORM9 are populated  If INLINEFORM10 , LiLi invokes the prediction model for INLINEFORM11  
Lifelong Relation Prediction  Given a relation INLINEFORM0 , LiLi uses INLINEFORM1 and INLINEFORM2 (see INLINEFORM3 ) to train a prediction model (say, INLINEFORM4 ) with parameters INLINEFORM5   For a unknown INLINEFORM6 , the clue instances get stored in INLINEFORM7 and INLINEFORM8   Thus, LiLi populates INLINEFORM9 by taking 10% (see §4) of the instances from INLINEFORM10 and starts the training  For INLINEFORM11 , LiLi uses a LSTM BIBREF40 to compose the vector representation of each feature INLINEFORM12 as INLINEFORM13 and vector representation of INLINEFORM14 as INLINEFORM15   Next, LiLi computes the prediction value, INLINEFORM16 as sigmoid of the mean cosine similarity of all features and INLINEFORM17 , given by INLINEFORM18 ) and maximize the log-likelihood of INLINEFORM19 for training  Once INLINEFORM20 is trained, LiLi updates INLINEFORM21 [ INLINEFORM22 ] using INLINEFORM23   We also train an inverse model for INLINEFORM24 , INLINEFORM25 by reversing the path features in INLINEFORM26 and INLINEFORM27 which help in lifelong learning (discussed below)  Unlike BIBREF20 , BIBREF21 , while predicting the label for INLINEFORM28 , we compute a relation-specific prediction threshold INLINEFORM29 corresponding to INLINEFORM30 using INLINEFORM31 as: INLINEFORM32 and infer INLINEFORM33 as +ve if INLINEFORM34 and -ve otherwise  Here, INLINEFORM35 ( INLINEFORM36 ) is the mean prediction value for all +ve (-ve) examples in INLINEFORM37  
Models trained on a few examples (e g , clues acquired for unknown INLINEFORM0 ) with randomly initialized weights often perform poorly due to underfitting  Thus, we transfer knowledge (weights) from the past most similar (wrt INLINEFORM1 ) task in a lifelong learning manner BIBREF31   LiLi uses INLINEFORM2 to find the past most similar task for INLINEFORM3 as follows: LiLi computes trancated SVD of INLINEFORM4 as INLINEFORM5 and then, the similarity matrix INLINEFORM6   INLINEFORM7 provides the similarity between relations INLINEFORM8 and INLINEFORM9 in INLINEFORM10   Thus, LiLi chooses a source relation INLINEFORM11 to transfer weights  Here, INLINEFORM12 is the set of all INLINEFORM13 and INLINEFORM14 for which LiLi has already learned a prediction model  Now, if INLINEFORM15 or INLINEFORM16 , LiLi randomly initializes the weights INLINEFORM17 for INLINEFORM18 and proceeds with the training  Otherwise, LiLi uses INLINEFORM19 as initial weights and fine-tunes INLINEFORM20 with a low learning rate 
A Running Example  Considering the example shown in Figure 1, LiLi works as follows: first, LiLi executes INLINEFORM0 and detects that the source entity “Obama" and query relation “CitizenOf" are unknown  Thus, LiLi executes INLINEFORM1 to acquire clue (SF1) for “CitizenOf" and pushes the clue (+ve example) and two generated -ve examples into INLINEFORM2   Once the clues are processed and a prediction model is trained for “CitizenOf" by formulating separate strategies for them, LiLi becomes aware of “CitizenOf"  Now, as the clues have already been popped from INLINEFORM3 , the query becomes INLINEFORM4 and the strategy formulation process for the query resumes  Next, LiLi asks user to provide a connecting link for “Obama" by performing INLINEFORM5   Now, the query entities and relation being known, LiLi enumerates paths between “Obama" and “USA" by performing INLINEFORM6   Let an extracted path be “ INLINEFORM7 " with missing link between ( INLINEFORM8 , INLINEFORM9 )  LiLi asks the user to fill the link by performing INLINEFORM10 and then, extracts the complete feature “ INLINEFORM11 "  The feature set is then fed to the prediction model and inference is made as a result of INLINEFORM12   Thus, the formulated inference strategy is: “ INLINEFORM13 " 
Experiments
We now evaluate LiLi in terms of its predictive performance and strategy formulation abilities 
Data: We use two standard datasets (see Table 4): (1) Freebase FB15k, and (2) WordNet INLINEFORM0   Using each dataset, we build a fairly large graph and use it as the original KB ( INLINEFORM1 ) for evaluation  We also augment INLINEFORM2 with inverse triples ( INLINEFORM3 , INLINEFORM4 , INLINEFORM5 ) for each ( INLINEFORM6 , INLINEFORM7 , INLINEFORM8 ) following existing KBC methods 
Parameter Settings  Unless specified, the empirically set parameters (see Table 1) of LiLi are: INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 , INLINEFORM8 , INLINEFORM9 , INLINEFORM10   For training RL-model with INLINEFORM11 -greedy strategy, we use INLINEFORM12 , INLINEFORM13 , pre-training steps=50000  We used Keras deep learning library to implement and train the prediction model  We set batch-size as 128, max  training epoch as 150, dropout as 0 2, hidden units and embedding size as 300 and learning rate as 5e-3 which is reduced gradually on plateau with factor 0 5 and patience 5  Adam optimizer and early stopping were used in training  We also shuffle INLINEFORM14 in each epoch and adjust class weights inversely proportional to class frequencies in INLINEFORM15  
Labeled Dataset Generation and Simulated User Creation  We create a simulated user for each KB to evaluate LiLi  We create the labeled datasets, the simulated user’s knowledge base ( INLINEFORM0 ), and the base KB ( INLINEFORM1 ) from INLINEFORM2   INLINEFORM3 used as the initial KB graph ( INLINEFORM4 ) of LiLi 
We followed BIBREF16 for labeled dataset generation  For Freebase, we found 86 relations with INLINEFORM0 triples and randomly selected 50 from various domains  We randomly shuffle the list of 50 relations, select 25% of them as unknown relations and consider the rest (75%) as known relations  For each known relation INLINEFORM1 , we randomly shuffle the list of distinct triples for INLINEFORM2 , choose 1000 triples and split them into 60% training, 10% validation and 20% test  Rest 10% along with the leftover (not included in the list of 1000) triples are added to INLINEFORM3   For each unknown relation INLINEFORM4 , we remove all triples of INLINEFORM5 from INLINEFORM6 and add them to INLINEFORM7   In this process, we also randomly choose 20% triples as test instances for unknown INLINEFORM8 which are excluded from INLINEFORM9   Note that, now INLINEFORM10 has at least 10% of chosen triples for each INLINEFORM11 (known and unknown) and so, user is always able to provide clues for both cases  For each labeled dataset, we randomly choose 10% of the entities present in dataset triples, remove triples involving those entities from INLINEFORM12 and add to INLINEFORM13   At this point, INLINEFORM14 gets reduced to INLINEFORM15 and is used as INLINEFORM16 for LiLi  The dataset stats in Table 4 shows that the base KB (60% triples of INLINEFORM17 ) is highly sparse (compared to original KB) which makes the inference task much harder  WordNet dataset being small, we select all 18 relations for evaluation and create labeled dataset, INLINEFORM18 and INLINEFORM19 following Freebase  Although the user may provide clues 100% of the time, it often cannot respond to MLQs and CLQs (due to lack of required triples/facts)  Thus, we further enrich INLINEFORM20 with external KB triples 
Given a relation INLINEFORM0 and an observed triple ( INLINEFORM1 , INLINEFORM2 , INLINEFORM3 ) in training or testing, the pair ( INLINEFORM4 , INLINEFORM5 ) is regarded as a +ve instance for INLINEFORM6   Following BIBREF18 , for each +ve instance ( INLINEFORM7 , INLINEFORM8 ), we generate two negative ones, one by randomly corrupting the source INLINEFORM9 , and the other by corrupting the target INLINEFORM10   Note that, the test triples are not in INLINEFORM11 or INLINEFORM12 and none of the -ve instances overlap with the +ve ones 
Baselines  As none of the existing KBC methods can solve the OKBC problem, we choose various versions of LiLi as baselines 
Single: Version of LiLi where we train a single prediction model INLINEFORM0 for all test relations 
Sep: We do not transfer (past learned) weights for initializing INLINEFORM0 , i e , we disable LL 
F-th): Here, we use a fixed prediction threshold 0 5 instead of relation-specific threshold INLINEFORM0  
BG: The missing or connecting links (when the user does not respond) are filled with “@-RelatedTo-@" blindly, no guessing mechanism 
w/o PTS: LiLi does not ask for additional clues via past task selection for skillset improvement 
Evaluation Metrics  To evaluate the strategy formulation ability, we introduce a measure called Coverage( INLINEFORM0 ), defined as the fraction of total query data instances, for which LiLi has successfully formulated strategies that lead to winning  If LiLi wins on all episodes for a given dataset, INLINEFORM1 is 1 0  To evaluate the predictive performance, we use Avg  MCC and avg  +ve F1 score 
Results and Analysis
Evaluation-I: Strategy Formulation Ability  Table 5 shows the list of inference strategies formulated by LiLi for various INLINEFORM0 and INLINEFORM1 , which control the strategy formulation of LiLi  When INLINEFORM2 , LiLi cannot interact with user and works like a closed-world method  Thus, INLINEFORM3 drops significantly (0 47)  When INLINEFORM4 , i e  with only one interaction per query, LiLi acquires knowledge well for instances where either of the entities or relation is unknown  However, as one unknown entity may appear in multiple test triples, once the entity becomes known, LiLi doesn’t need to ask for it again and can perform inference on future triples causing significant increase in INLINEFORM5 (0 97)  When INLINEFORM6 , LiLi is able to perform inference on all instances and INLINEFORM7 becomes 1  For INLINEFORM8 , LiLi uses INLINEFORM9 only once (as only one MLQ satisfies INLINEFORM10 ) compared to INLINEFORM11   In summary, LiLi’s RL-model can effectively formulate query-specific inference strategies (based on specified parameter values)  Evaluation-II: Predictive Performance  Table 6 shows the comparative performance of LiLi with baselines  To judge the overall improvements, we performed paired t-test considering +ve F1 scores on each relation as paired data  Considering both KBs and all relation types, LiLi outperforms Sep with INLINEFORM12   If we set INLINEFORM13 (training with very few clues), LiLi outperforms Sep with INLINEFORM14 on Freebase considering MCC  Thus, the lifelong learning mechanism is effective in transferring helpful knowledge  Single model performs better than Sep for unknown relations due to the sharing of knowledge (weights) across tasks  However, for known relations, performance drops because, as a new relation arrives to the system, old weights get corrupted and catastrophic forgetting occurs  For unknown relations, as the relations are evaluated just after training, there is no chance for catastrophic forgetting  The performance improvement ( INLINEFORM15 ) of LiLi over F-th on Freebase signifies that the relation-specific threshold INLINEFORM16 works better than fixed threshold 0 5 because, if all prediction values for test instances lie above (or below) 0 5, F-th predicts all instances as +ve (-ve) which degrades its performance  Due to the utilization of contextual similarity (highly correlated with class labels) of entity-pairs, LiLi’s guessing mechanism works better ( INLINEFORM17 ) than blind guessing (BG)  The past task selection mechanism of LiLi also improves its performance over w/o PTS, as it acquires more clues during testing for poorly performed tasks (evaluated on validation set)  For Freebase, due to a large number of past tasks [9 (25% of 38)], the performance difference is more significant ( INLINEFORM18 )  For WordNet, the number is relatively small [3 (25% of 14)] and hence, the difference is not significant 
Evaluation-III: User Interaction vs  Performance  Table 7 shows the results of LiLi by varying clue acquisition rate ( INLINEFORM0 )  We use Freebase for tuning INLINEFORM1 due to its higher number of unknown test relations compared to WordNet  LiLi’s performance improves significantly as it acquires more clues from the user  The results on INLINEFORM2 outperforms ( INLINEFORM3 ) that on INLINEFORM4   Table 8 shows the results of LiLi on user responses to MLQ’s and CLQ’s  Answering MLQ’s and CLQ’s is very hard for simulated users (unlike crowd-sourcing) as often INLINEFORM5 lacks the required triple  Thus, we attempt to analyze how the performance is effected if the user does not respond at all  The results show a clear trend in overall performance improvement when the user responds  However, the improvement is not significant as the simulated user’s query satisfaction rate (1% MLQs and 10% CLQs) is very small  But, the analysis shows the effectiveness of LiLi’s guessing mechanism and continual learning ability that help in achieving avg  +ve F1 of 0 57 and 0 62 on FB and WN respectively with minimal participation of the user 
Conclusion
In this paper, we are interested in building a generic engine for continuous knowledge learning in human-machine conversations  We first showed that the problem underlying the engine can be formulated as an open-world knowledge base completion (OKBC) problem  We then proposed an lifelong interactive learning and inference (LiLi) approach to solving the OKBC problem  OKBC is a generalization of KBC  LiLi solves the OKBC problem by first formulating a query-specific inference strategy using RL and then executing it to solve the problem by interacting with the user in a lifelong learning manner  Experimental results showed the effectiveness of LiLi in terms of both predictive quality and strategy formulation ability  We believe that a system with the LiLi approach can serve as a knowledge learning engine for conversations  Our future work will improve LiLi to make more accurate 
Acknowledgments
This work was supported in part by National Science Foundation (NSF) under grant no  IIS-1407927 and IIS-1650900, and a gift from Huawei Technologies Co Ltd 

ground_truth: Coverage, Avg. MCC and avg. +ve F1 score

strategy formulation ability, we introduce a measure called Coverage( INLINEFORM0 ), To evaluate the predictive performance, we use Avg. MCC and avg. +ve F1 score

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What sentiment classification dataset is used?
Context 2: Introduction
Recurrent neural networks (RNNs), including gated variants such as the long short-term memory (LSTM) BIBREF0 have become the standard model architecture for deep learning approaches to sequence modeling tasks  RNNs repeatedly apply a function with trainable parameters to a hidden state  Recurrent layers can also be stacked, increasing network depth, representational power and often accuracy  RNN applications in the natural language domain range from sentence classification BIBREF1 to word- and character-level language modeling BIBREF2   RNNs are also commonly the basic building block for more complex models for tasks such as machine translation BIBREF3 , BIBREF4 , BIBREF5 or question answering BIBREF6 , BIBREF7   Unfortunately standard RNNs, including LSTMs, are limited in their capability to handle tasks involving very long sequences, such as document classification or character-level machine translation, as the computation of features or states for different parts of the document cannot occur in parallel 
Convolutional neural networks (CNNs) BIBREF8 , though more popular on tasks involving image data, have also been applied to sequence encoding tasks BIBREF9   Such models apply time-invariant filter functions in parallel to windows along the input sequence  CNNs possess several advantages over recurrent models, including increased parallelism and better scaling to long sequences such as those often seen with character-level language data  Convolutional models for sequence processing have been more successful when combined with RNN layers in a hybrid architecture BIBREF10 , because traditional max- and average-pooling approaches to combining convolutional features across timesteps assume time invariance and hence cannot make full use of large-scale sequence order information 
We present quasi-recurrent neural networks for neural sequence modeling  QRNNs address both drawbacks of standard models: like CNNs, QRNNs allow for parallel computation across both timestep and minibatch dimensions, enabling high throughput and good scaling to long sequences  Like RNNs, QRNNs allow the output to depend on the overall order of elements in the sequence  We describe QRNN variants tailored to several natural language tasks, including document-level sentiment classification, language modeling, and character-level machine translation  These models outperform strong LSTM baselines on all three tasks while dramatically reducing computation time 
Model
Each layer of a quasi-recurrent neural network consists of two kinds of subcomponents, analogous to convolution and pooling layers in CNNs  The convolutional component, like convolutional layers in CNNs, allows fully parallel computation across both minibatches and spatial dimensions, in this case the sequence dimension  The pooling component, like pooling layers in CNNs, lacks trainable parameters and allows fully parallel computation across minibatch and feature dimensions 
Given an input sequence INLINEFORM0 of INLINEFORM1 INLINEFORM2 -dimensional vectors INLINEFORM3 , the convolutional subcomponent of a QRNN performs convolutions in the timestep dimension with a bank of INLINEFORM4 filters, producing a sequence INLINEFORM5 of INLINEFORM6 -dimensional candidate vectors INLINEFORM7   In order to be useful for tasks that include prediction of the next token, the filters must not allow the computation for any given timestep to access information from future timesteps  That is, with filters of width INLINEFORM8 , each INLINEFORM9 depends only on INLINEFORM10 through INLINEFORM11   This concept, known as a masked convolution BIBREF11 , is implemented by padding the input to the left by the convolution's filter size minus one 
We apply additional convolutions with separate filter banks to obtain sequences of vectors for the elementwise gates that are needed for the pooling function  While the candidate vectors are passed through a INLINEFORM0 nonlinearity, the gates use an elementwise sigmoid  If the pooling function requires a forget gate INLINEFORM1 and an output gate INLINEFORM2 at each timestep, the full set of computations in the convolutional component is then: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , each in INLINEFORM3 , are the convolutional filter banks and INLINEFORM4 denotes a masked convolution along the timestep dimension  Note that if the filter width is 2, these equations reduce to the LSTM-like DISPLAYFORM0
Convolution filters of larger width effectively compute higher INLINEFORM0 -gram features at each timestep; thus larger widths are especially important for character-level tasks 
Suitable functions for the pooling subcomponent can be constructed from the familiar elementwise gates of the traditional LSTM cell  We seek a function controlled by gates that can mix states across timesteps, but which acts independently on each channel of the state vector  The simplest option, which BIBREF12 term “dynamic average pooling”, uses only a forget gate: DISPLAYFORM0
We term these three options f-pooling, fo-pooling, and ifo-pooling respectively; in each case we initialize INLINEFORM0 or INLINEFORM1 to zero  Although the recurrent parts of these functions must be calculated for each timestep in sequence, their simplicity and parallelism along feature dimensions means that, in practice, evaluating them over even long sequences requires a negligible amount of computation time 
A single QRNN layer thus performs an input-dependent pooling, followed by a gated linear combination of convolutional features  As with convolutional neural networks, two or more QRNN layers should be stacked to create a model with the capacity to approximate more complex functions 
Variants
Motivated by several common natural language tasks, and the long history of work on related architectures, we introduce several extensions to the stacked QRNN described above  Notably, many extensions to both recurrent and convolutional models can be applied directly to the QRNN as it combines elements of both model types 
Regularization An important extension to the stacked QRNN is a robust regularization scheme inspired by recent work in regularizing LSTMs 
The need for an effective regularization method for LSTMs, and dropout's relative lack of efficacy when applied to recurrent connections, led to the development of recurrent dropout schemes, including variational inference–based dropout BIBREF13 and zoneout BIBREF14   These schemes extend dropout to the recurrent setting by taking advantage of the repeating structure of recurrent networks, providing more powerful and less destructive regularization 
Variational inference–based dropout locks the dropout mask used for the recurrent connections across timesteps, so a single RNN pass uses a single stochastic subset of the recurrent weights  Zoneout stochastically chooses a new subset of channels to “zone out” at each timestep; for these channels the network copies states from one timestep to the next without modification 
As QRNNs lack recurrent weights, the variational inference approach does not apply  Thus we extended zoneout to the QRNN architecture by modifying the pooling function to keep the previous pooling state for a stochastic subset of channels  Conveniently, this is equivalent to stochastically setting a subset of the QRNN's INLINEFORM0 gate channels to 1, or applying dropout on INLINEFORM1 : DISPLAYFORM0
Thus the pooling function itself need not be modified at all  We note that when using an off-the-shelf dropout layer in this context, it is important to remove automatic rescaling functionality from the implementation if it is present  In many experiments, we also apply ordinary dropout between layers, including between word embeddings and the first QRNN layer 
Densely-Connected Layers We can also extend the QRNN architecture using techniques introduced for convolutional networks  For sequence classification tasks, we found it helpful to use skip-connections between every QRNN layer, a technique termed “dense convolution” by BIBREF15   Where traditional feed-forward or convolutional networks have connections only between subsequent layers, a “DenseNet” with INLINEFORM0 layers has feed-forward or convolutional connections between every pair of layers, for a total of INLINEFORM1   This can improve gradient flow and convergence properties, especially in deeper networks, although it requires a parameter count that is quadratic in the number of layers 
When applying this technique to the QRNN, we include connections between the input embeddings and every QRNN layer and between every pair of QRNN layers  This is equivalent to concatenating each QRNN layer's input to its output along the channel dimension before feeding the state into the next layer  The output of the last layer alone is then used as the overall encoding result 
Encoder–Decoder Models To demonstrate the generality of QRNNs, we extend the model architecture to sequence-to-sequence tasks, such as machine translation, by using a QRNN as encoder and a modified QRNN, enhanced with attention, as decoder  The motivation for modifying the decoder is that simply feeding the last encoder hidden state (the output of the encoder's pooling layer) into the decoder's recurrent pooling layer, analogously to conventional recurrent encoder–decoder architectures, would not allow the encoder state to affect the gate or update values that are provided to the decoder's pooling layer  This would substantially limit the representational power of the decoder 
Instead, the output of each decoder QRNN layer's convolution functions is supplemented at every timestep with the final encoder hidden state  This is accomplished by adding the result of the convolution for layer INLINEFORM0 (e g , INLINEFORM1 , in INLINEFORM2 ) with broadcasting to a linearly projected copy of layer INLINEFORM3 's last encoder state (e g , INLINEFORM4 , in INLINEFORM5 ): DISPLAYFORM0
where the tilde denotes that INLINEFORM0 is an encoder variable  Encoder–decoder models which operate on long sequences are made significantly more powerful with the addition of soft attention BIBREF3 , which removes the need for the entire input representation to fit into a fixed-length encoding vector  In our experiments, we computed an attentional sum of the encoder's last layer's hidden states  We used the dot products of these encoder hidden states with the decoder's last layer's un-gated hidden states, applying a INLINEFORM1 along the encoder timesteps, to weight the encoder states into an attentional sum INLINEFORM2 for each decoder timestep  This context, and the decoder state, are then fed into a linear layer followed by the output gate: DISPLAYFORM0
where INLINEFORM0 is the last layer 
While the first step of this attention procedure is quadratic in the sequence length, in practice it takes significantly less computation time than the model's linear and convolutional layers due to the simple and highly parallel dot-product scoring function 
Experiments
We evaluate the performance of the QRNN on three different natural language tasks: document-level sentiment classification, language modeling, and character-based neural machine translation  Our QRNN models outperform LSTM-based models of equal hidden size on all three tasks while dramatically improving computation speed  Experiments were implemented in Chainer BIBREF16  
Sentiment Classification
We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset BIBREF17   The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words BIBREF18   We compare only to other results that do not make use of additional unlabeled data (thus excluding e g , BIBREF19 ) 
Our best performance on a held-out development set was achieved using a four-layer densely-connected QRNN with 256 units per layer and word vectors initialized using 300-dimensional cased GloVe embeddings BIBREF20   Dropout of 0 3 was applied between layers, and we used INLINEFORM0 regularization of INLINEFORM1   Optimization was performed on minibatches of 24 examples using RMSprop BIBREF21 with learning rate of INLINEFORM2 , INLINEFORM3 , and INLINEFORM4  
Small batch sizes and long sequence lengths provide an ideal situation for demonstrating the QRNN's performance advantages over traditional recurrent architectures  We observed a speedup of 3 2x on IMDb train time per epoch compared to the optimized LSTM implementation provided in NVIDIA's cuDNN library  For specific batch sizes and sequence lengths, a 16x speed gain is possible  Figure FIGREF15 provides extensive speed comparisons 
In Figure FIGREF12 , we visualize the hidden state vectors INLINEFORM0 of the final QRNN layer on part of an example from the IMDb dataset  Even without any post-processing, changes in the hidden state are visible and interpretable in regards to the input  This is a consequence of the elementwise nature of the recurrent pooling function, which delays direct interaction between different channels of the hidden state until the computation of the next QRNN layer 
Language Modeling
We replicate the language modeling experiment of BIBREF2 and BIBREF13 to benchmark the QRNN architecture for natural language sequence prediction  The experiment uses a standard preprocessed version of the Penn Treebank (PTB) by BIBREF25  
We implemented a gated QRNN model with medium hidden size: 2 layers with 640 units in each layer  Both QRNN layers use a convolutional filter width INLINEFORM0 of two timesteps  While the “medium” models used in other work BIBREF2 , BIBREF13 consist of 650 units in each layer, it was more computationally convenient to use a multiple of 32  As the Penn Treebank is a relatively small dataset, preventing overfitting is of considerable importance and a major focus of recent research  It is not obvious in advance which of the many RNN regularization schemes would perform well when applied to the QRNN  Our tests showed encouraging results from zoneout applied to the QRNN's recurrent pooling layer, implemented as described in Section SECREF5  
The experimental settings largely followed the “medium” setup of BIBREF2   Optimization was performed by stochastic gradient descent (SGD) without momentum  The learning rate was set at 1 for six epochs, then decayed by 0 95 for each subsequent epoch, for a total of 72 epochs  We additionally used INLINEFORM0 regularization of INLINEFORM1 and rescaled gradients with norm above 10  Zoneout was applied by performing dropout with ratio 0 1 on the forget gates of the QRNN, without rescaling the output of the dropout function  Batches consist of 20 examples, each 105 timesteps 
Comparing our results on the gated QRNN with zoneout to the results of LSTMs with both ordinary and variational dropout in Table TABREF14 , we see that the QRNN is highly competitive  The QRNN without zoneout strongly outperforms both our medium LSTM and the medium LSTM of BIBREF2 which do not use recurrent dropout and is even competitive with variational LSTMs  This may be due to the limited computational capacity that the QRNN's pooling layer has relative to the LSTM's recurrent weights, providing structural regularization over the recurrence 
Without zoneout, early stopping based upon validation loss was required as the QRNN would begin overfitting  By applying a small amount of zoneout ( INLINEFORM0 ), no early stopping is required and the QRNN achieves competitive levels of perplexity to the variational LSTM of BIBREF13 , which had variational inference based dropout of 0 2 applied recurrently  Their best performing variation also used Monte Carlo (MC) dropout averaging at test time of 1000 different masks, making it computationally more expensive to run 
When training on the PTB dataset with an NVIDIA K40 GPU, we found that the QRNN is substantially faster than a standard LSTM, even when comparing against the optimized cuDNN LSTM  In Figure FIGREF15 we provide a breakdown of the time taken for Chainer's default LSTM, the cuDNN LSTM, and QRNN to perform a full forward and backward pass on a single batch during training of the RNN LM on PTB  For both LSTM implementations, running time was dominated by the RNN computations, even with the highly optimized cuDNN implementation  For the QRNN implementation, however, the “RNN” layers are no longer the bottleneck  Indeed, there are diminishing returns from further optimization of the QRNN itself as the softmax and optimization overhead take equal or greater time  Note that the softmax, over a vocabulary size of only 10,000 words, is relatively small; for tasks with larger vocabularies, the softmax would likely dominate computation time 
It is also important to note that the cuDNN library's RNN primitives do not natively support any form of recurrent dropout  That is, running an LSTM that uses a state-of-the-art regularization scheme at cuDNN-like speeds would likely require an entirely custom kernel 
Character-level Neural Machine Translation
We evaluate the sequence-to-sequence QRNN architecture described in SECREF5 on a challenging neural machine translation task, IWSLT German–English spoken-domain translation, applying fully character-level segmentation  This dataset consists of 209,772 sentence pairs of parallel training data from transcribed TED and TEDx presentations, with a mean sentence length of 103 characters for German and 93 for English  We remove training sentences with more than 300 characters in English or German, and use a unified vocabulary of 187 Unicode code points 
Our best performance on a development set (TED tst2013) was achieved using a four-layer encoder–decoder QRNN with 320 units per layer, no dropout or INLINEFORM0 regularization, and gradient rescaling to a maximum magnitude of 5  Inputs were supplied to the encoder reversed, while the encoder convolutions were not masked  The first encoder layer used convolutional filter width INLINEFORM1 , while the other encoder layers used INLINEFORM2   Optimization was performed for 10 epochs on minibatches of 16 examples using Adam BIBREF28 with INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , and INLINEFORM6   Decoding was performed using beam search with beam width 8 and length normalization INLINEFORM7   The modified log-probability ranking criterion is provided in the appendix 
Results using this architecture were compared to an equal-sized four-layer encoder–decoder LSTM with attention, applying dropout of 0 2  We again optimized using Adam; other hyperparameters were equal to their values for the QRNN and the same beam search procedure was applied  Table TABREF17 shows that the QRNN outperformed the character-level LSTM, almost matching the performance of a word-level attentional baseline 
Related Work
Exploring alternatives to traditional RNNs for sequence tasks is a major area of current research  Quasi-recurrent neural networks are related to several such recently described models, especially the strongly-typed recurrent neural networks (T-RNN) introduced by BIBREF12   While the motivation and constraints described in that work are different, BIBREF12 's concepts of “learnware” and “firmware” parallel our discussion of convolution-like and pooling-like subcomponents  As the use of a fully connected layer for recurrent connections violates the constraint of “strong typing”, all strongly-typed RNN architectures (including the T-RNN, T-GRU, and T-LSTM) are also quasi-recurrent  However, some QRNN models (including those with attention or skip-connections) are not “strongly typed”  In particular, a T-RNN differs from a QRNN as described in this paper with filter size 1 and f-pooling only in the absence of an activation function on INLINEFORM0   Similarly, T-GRUs and T-LSTMs differ from QRNNs with filter size 2 and fo- or ifo-pooling respectively in that they lack INLINEFORM1 on INLINEFORM2 and use INLINEFORM3 rather than sigmoid on INLINEFORM4  
The QRNN is also related to work in hybrid convolutional–recurrent models  BIBREF31 apply CNNs at the word level to generate INLINEFORM0 -gram features used by an LSTM for text classification  BIBREF32 also tackle text classification by applying convolutions at the character level, with a stride to reduce sequence length, then feeding these features into a bidirectional LSTM  A similar approach was taken by BIBREF10 for character-level machine translation  Their model's encoder uses a convolutional layer followed by max-pooling to reduce sequence length, a four-layer highway network, and a bidirectional GRU  The parallelism of the convolutional, pooling, and highway layers allows training speed comparable to subword-level models without hard-coded text segmentation 
The QRNN encoder–decoder model shares the favorable parallelism and path-length properties exhibited by the ByteNet BIBREF33 , an architecture for character-level machine translation based on residual convolutions over binary trees  Their model was constructed to achieve three desired properties: parallelism, linear-time computational complexity, and short paths between any pair of words in order to better propagate gradient signals 
Conclusion
Intuitively, many aspects of the semantics of long sequences are context-invariant and can be computed in parallel (e g , convolutionally), but some aspects require long-distance context and must be computed recurrently  Many existing neural network architectures either fail to take advantage of the contextual information or fail to take advantage of the parallelism  QRNNs exploit both parallelism and context, exhibiting advantages from both convolutional and recurrent neural networks  QRNNs have better predictive accuracy than LSTM-based models of equal hidden size, even though they use fewer parameters and run substantially faster  Our experiments show that the speed and accuracy advantages remain consistent across tasks and at both word and character levels 
Extensions to both CNNs and RNNs are often directly applicable to the QRNN, while the model's hidden states are more interpretable than those of other recurrent architectures as its channels maintain their independence across timesteps  We believe that QRNNs can serve as a building block for long-sequence tasks that were previously impractical with traditional RNNs 
Beam search ranking criterion
The modified log-probability ranking criterion we used in beam search for translation experiments is: DISPLAYFORM0
where INLINEFORM0 is a length normalization parameter BIBREF34 , INLINEFORM1 is the INLINEFORM2 th output character, and INLINEFORM3 is a “target length” equal to the source sentence length plus five characters  This reduces at INLINEFORM4 to ordinary beam search with probabilities: DISPLAYFORM0
and at INLINEFORM0 to beam search with probabilities normalized by length (up to the target length): DISPLAYFORM0
Conveniently, this ranking criterion can be computed at intermediate beam-search timesteps, obviating the need to apply a separate reranking on complete hypotheses 

ground_truth: the IMDb movie review dataset BIBREF17

IMDb movie review

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How much improvement is gained from Adversarial Reward Augmented Maximum Likelihood (ARAML)?
Context 3: Introduction
Natural text generation, as a key task in NLP, has been advanced substantially thanks to the flourish of neural models BIBREF0 , BIBREF1   Typical frameworks such as sequence-to-sequence (seq2seq) have been applied to various generation tasks, including machine translation BIBREF2 and dialogue generation BIBREF3   The standard paradigm to train such neural models is maximum likelihood estimation (MLE), which maximizes the log-likelihood of observing each word in the text given the ground-truth proceeding context BIBREF4  
Although widely used, MLE suffers from the exposure bias problem BIBREF5 , BIBREF6 : during test, the model sequentially predicts the next word conditioned on its previous generated words while during training conditioned on ground-truth words  To tackle this problem, generative adversarial networks (GAN) with reinforcement learning (RL) training approaches have been introduced to text generation tasks BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , where the discriminator is trained to distinguish real and generated text samples to provide reward signals for the generator, and the generator is optimized via policy gradient BIBREF7  
However, recent studies have shown that potential issues of training GANs on discrete data are more severe than exposure bias BIBREF14 , BIBREF15   One of the fundamental issues when generating discrete text samples with GANs is training instability  Updating the generator with policy gradient always leads to an unstable training process because it's difficult for the generator to derive positive and stable reward signals from the discriminator even with careful pre-training BIBREF8   As a result, the generator gets lost due to the high variance of reward signals and the training process may finally collapse BIBREF16  
In this paper, we propose a novel adversarial training framework called Adversarial Reward Augmented Maximum Likelihood (ARAML) to deal with the instability issue of training GANs for text generation  At each iteration of adversarial training, we first train the discriminator to assign higher rewards to real data than to generated samples  Then, inspired by reward augmented maximum likelihood (RAML) BIBREF17 , the generator is updated on the samples acquired from a stationary distribution with maximum likelihood estimation (MLE), weighted by the discriminator's rewards  This stationary distribution is designed to guarantee that training samples are surrounding the real data, thus the exploration space of our generator is indeed restricted by the MLE training objective, resulting in more stable training  Compared to other text GANs with RL training techniques, our framework acquires samples from the stationary distribution rather than the generator's distribution, and uses RAML training paradigm to optimize the generator instead of policy gradient  Our contributions are mainly as follows:
Related Work
Recently, text generation has been widely studied with neural models trained with maximum likelihood estimation BIBREF4   However, MLE tends to generate universal text BIBREF18   Various methods have been proposed to enhance the generation quality by refining the objective function BIBREF18 , BIBREF19 or modifying the generation distribution with external information like topic BIBREF20 , sentence type BIBREF21 , emotion BIBREF22 and knowledge BIBREF23  
As mentioned above, MLE suffers from the exposure bias problem BIBREF5 , BIBREF6   Thus, reinforcement learning has been introduced to text generation tasks such as policy gradient BIBREF6 and actor-critic BIBREF24   BIBREF17 proposed an efficient and stable approach called Reward Augmented Maximum Likelihood (RAML), which connects the log-likelihood and expected rewards to incorporate MLE training objective into RL framework 
Since some text generation tasks have no explicit metrics to be directly optimized, adversarial training has been applied to generating discrete text samples with a discriminator to learn a proper reward  For instance, SeqGAN BIBREF7 devised a discriminator to distinguish the real data and generated samples, and a generator to maximize the reward from the discriminator via policy gradient  Other variants of GANs have been proposed to improve the generator or the discriminator  To improve the generator, MaliGAN BIBREF8 developed a normalized maximum likelihood optimization target for the generator to stably model the discrete sequences  LeakGAN BIBREF11 guided the generator with reward signals leaked from the discriminator at all generation steps to deal with long text generation task  MaskGAN BIBREF10 employed an actor-critic architecture to make the generator fill in missing text conditioned on the surrounding context, which is expected to mitigate the problem of mode collapse  As for the discriminator, RankGAN BIBREF9 replaced traditional discriminator with a ranker to learn the relative ranking information between the real texts and generated ones  Inverse reinforcement learning BIBREF12 used a trainable reward approximator as the discriminator to provide dense reward signals at each generation step  DPGAN BIBREF13 introduced a language model based discriminator and regarded cross-entropy as rewards to promote the diversity of generation results 
The most similar works to our model are RAML BIBREF17 and MaliGAN BIBREF8 : 1) Compared with RAML, our model adds a discriminator to learn the reward signals instead of choosing existing metrics as rewards  We believe that our model can adapt to various text generation tasks, particularly those without explicit evaluation metrics  2) Unlike MaliGAN, we acquire samples from a fixed distribution near the real data rather than the generator's distribution, which is expected to make the training process more stable 
Task Definition and Model Overview
Text generation can be formulated as follows: given the real data distribution INLINEFORM0 , the task is to train a generative model INLINEFORM1 where INLINEFORM2 can fit INLINEFORM3 well  In this formulation, INLINEFORM4 and INLINEFORM5 denotes a word in the vocabulary INLINEFORM6  
Figure FIGREF3 shows the overview of our model ARAML  This adversarial training framework consists of two phases: 1) The discriminator is trained to assign higher rewards to real data than to generated data  2) The generator is trained on the samples acquired from a stationary distribution with reward augmented MLE training objective  This training paradigm of the generator indeed constrains the search space with the MLE training objective, which alleviates the issue of unstable training 
Discriminator
The discriminator INLINEFORM0 aims to distinguish real data and generated data like other GANs  Inspired by Least-Square GAN BIBREF25 , we devise the loss function as follows: DISPLAYFORM0
This loss function forces the discriminator to assign higher rewards to real data than to generated data, so the discriminator can learn to provide more proper rewards as the training proceeds 
Generator
The training objective of our generator INLINEFORM0 is derived from the objective of other discrete GANs with RL training method: DISPLAYFORM0
where INLINEFORM0 denotes the rewards from the discriminator INLINEFORM1 and the entropy regularized term INLINEFORM2 encourages INLINEFORM3 to generate diverse text samples  INLINEFORM4 is a temperature hyper-parameter to balance these two terms 
As mentioned above, discrete GANs suffer from the instability issue due to policy gradient, thus they are consequently difficult to train  Inspired by RAML BIBREF17 , we introduce an exponential payoff distribution INLINEFORM0 to connect RL loss with RAML loss: DISPLAYFORM0
where INLINEFORM0   Thus, we can rewrite INLINEFORM1 with INLINEFORM2 and INLINEFORM3 as follows: DISPLAYFORM0
Following RAML, we remove the constant term and optimize the KL divergence in the opposite direction: DISPLAYFORM0
where INLINEFORM0 is a constant in the training phase of the generator  It has been proved that INLINEFORM1 and INLINEFORM2 are equivalent up to their first order Taylor approximations, and they have the same global optimum BIBREF17   INLINEFORM3 can be trained in a MLE-like fashion but sampling from the distribution INLINEFORM4 is intractable in the adversarial setting, because INLINEFORM5 varies with the discriminator INLINEFORM6   Thus, we introduce importance sampling to separate sampling process from INLINEFORM7 and obtain the final loss function: DISPLAYFORM0
where INLINEFORM0 denotes a stationary distribution and INLINEFORM1   To optimize this loss function, we first construct the fixed distribution INLINEFORM2 to get samples, and devise the proper reward function INLINEFORM3 to train the generator in a stable and effective way 
We construct the distribution INLINEFORM0 based on INLINEFORM1 : DISPLAYFORM0
In this way, INLINEFORM0 can be designed to guarantee that INLINEFORM1 is near INLINEFORM2 , leading to a more stable training process  To obtain a new sample INLINEFORM3 from a real data sample INLINEFORM4 , we can design three steps which contain sampling an edit distance INLINEFORM5 , the positions INLINEFORM6 for substitution and the new words INLINEFORM7 filled into the corresponding positions  Thus, INLINEFORM8 can be decomposed into three terms: DISPLAYFORM0
The first step is to sample an edit distance based on a real data sample INLINEFORM0 , where INLINEFORM1 is a sequence of length INLINEFORM2   The number of sentences which have the edit distance INLINEFORM3 to some input sentence can be computed approximately as below: DISPLAYFORM0
where INLINEFORM0 denotes the number of sentences which have an edit distance INLINEFORM1 to a sentence of length INLINEFORM2 , and INLINEFORM3 indicates the size of vocabulary  We then follow BIBREF17 to re-scale the counts by INLINEFORM4 and do normalization, so that we can sample an edit distance INLINEFORM5 from: DISPLAYFORM0
where INLINEFORM0 , as a temperature hyper-parameter, restricts the search space surrounding the original sentence  Larger INLINEFORM1 brings more samples with long edit distances 
The next step is to select positions for substitution based on the sampled edit distance INLINEFORM0   Intuitively, we can randomly choose INLINEFORM1 distinct positions in INLINEFORM2 to be replaced by new words  The probability of choosing the position INLINEFORM3 is calculated as follows: DISPLAYFORM0
Following this sampling strategy, we can obtain the position set INLINEFORM0   This strategy approximately guarantees that the edit distance between a new sentence and the original sentence is INLINEFORM1  
At the final step, our model determines new words for substitution at each sampled position INLINEFORM0   We can formulate this sampling process from the original sequence INLINEFORM1 to a new sample INLINEFORM2 as a sequential transition INLINEFORM3   At each step from INLINEFORM4 to INLINEFORM5 INLINEFORM6 , we first sample a new word INLINEFORM7 from the distribution INLINEFORM8 , then replace the old word at position INLINEFORM9 of INLINEFORM10 to obtain INLINEFORM11   The whole sampling process can be decomposed as follows: DISPLAYFORM0
There are two common sampling strategies to model INLINEFORM0 , i e  random sampling and constrained sampling  Random sampling strategy samples a new word INLINEFORM1 according to the uniform distribution over the vocabulary INLINEFORM2 BIBREF17 , while constrained sampling strategy samples INLINEFORM3 to maximize the language model score of the target sentence INLINEFORM4 BIBREF26 , BIBREF27   Here, we adopt constrained sampling in our model and compare the performances of two strategies in the experiment 
We devise the reward function INLINEFORM0 according to the discriminator's output INLINEFORM1 and the stationary distribution INLINEFORM2 : DISPLAYFORM0
Intuitively, this reward function encourages the generator to generate sentences with large sampling probability and high rewards from the discriminator  Thus, the weight of samples INLINEFORM0 can be calculated as follows: DISPLAYFORM0
So far, we can successfully optimize the generator's loss INLINEFORM0 via Equation EQREF12   This training paradigm makes our generator avoid possible variances caused by policy gradient and get more stable reward signals from the discriminator, because our generator is restricted to explore the training samples near the real data 
[htb] Adversarial Reward Augmented Maximum Likelihood [1]
Total adversarial training iterations: INLINEFORM0
Steps of training generator: INLINEFORM0
Steps of training discriminator: INLINEFORM0
Pre-train the generator INLINEFORM0 with MLE loss Generate samples from INLINEFORM1 Pre-train the discriminator INLINEFORM2 via Eq ( EQREF6 ) Construct INLINEFORM3 via Eq ( EQREF14 ) - Eq ( EQREF19 ) each INLINEFORM4 each INLINEFORM5 Update INLINEFORM6 via Eq ( EQREF12 ) each INLINEFORM7 Update INLINEFORM8 via Eq ( EQREF6 )
Extension to Conditional Text Generation
We have shown our adversarial training framework for text generation tasks without an input  Actually, it can also be extended to conditional text generation tasks like dialogue generation  Given the data distribution INLINEFORM0 where INLINEFORM1 denote contexts and responses respectively, the objective function of ARAML's generator can be modified as below: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 is trained to distinguish whether INLINEFORM2 is the true response to INLINEFORM3  
Comparison with RAML and MaliGAN
The most similar works to our framework are RAML BIBREF17 and MaliGAN BIBREF8   The main difference among them is the training objective of their generators  We have shown different objective functions in Table TABREF26   For comparison, we use the form with no input for all the three models 
Our model is greatly inspired by RAML, which gets samples from a non-parametric distribution INLINEFORM0 constructed based on a specific reward  Compared to RAML, our reward comes from a learnable discriminator which varies as the adversarial training proceeds rather than a specific reward function  This difference equips our framework with the ability to adapt to the text generation tasks with no explicit evaluation metrics as rewards 
Our model is also similar to MaliGAN, which gets samples from the generator's distribution  In MaliGAN's training objective, INLINEFORM0 also indicates the generator's distribution but it's used in the sampling phase and fixed at each optimization step  The weight of samples INLINEFORM1   Different from our model, MaliGAN acquires samples from the generator's distribution INLINEFORM2 , which usually brings samples with low rewards even with careful pre-training for the generator, leading to training instability  Instead, our framework gets samples from a stationary distribution INLINEFORM3 around real data, thus our training process is more stable 
Datasets
We evaluated ARAML on three datasets: COCO image caption dataset BIBREF28 , EMNLP2017 WMT dataset and WeiboDial single-turn dialogue dataset BIBREF29   COCO and EMNLP2017 WMT are the common benchmarks with no input to evaluate the performance of discrete GANs, and we followed the existing works to preprocess these datasets BIBREF12 , BIBREF11   WeiboDial, as a dialogue dataset, was applied to test the performance of our model with input trigger  We simply removed post-response pairs containing low-frequency words and randomly selected a subset for our training/test set  The statistics of three datasets are presented in Table TABREF28  
Baselines
We compared our model with MLE, RL and GAN baselines  Since COCO and EMNLP2017 WMT don't have input while WeiboDial regards posts as input, we chose the following baselines respectively:
MLE: a RNN model trained with MLE objective BIBREF4   Its extension, Seq2Seq, can work on the dialogue dataset BIBREF2  
SeqGAN: The first text GAN model that updates the generator with policy gradient based on the rewards from the discriminator BIBREF7  
LeakGAN: A variant of SeqGAN that provides rewards based on the leaked information of the discriminator for the generator BIBREF11  
MaliGAN: A variant of SeqGAN that optimizes the generator with a normalized maximum likelihood objective BIBREF8  
IRL: This inverse reinforcement learning method replaces the discriminator with a reward approximator to provide dense rewards BIBREF12  
RAML: A RL approach to incorporate MLE objective into RL training framework, which regards BLEU as rewards BIBREF17  
DialogGAN: An extension of SeqGAN tuned to dialogue generation task with MLE objective added to the adversarial objective BIBREF16  
DPGAN: A variant of DialogGAN which uses a language model based discriminator and regards cross-entropy as rewards BIBREF13  
Note that MLE, SeqGAN, LeakGAN, MaliGAN and IRL are the baselines on COCO and EMNLP2017 WMT, while MLE, RAML, DialogGAN, and DPGAN on WeiboDial  The original codes are used to test the baselines 
Implementation Details
The implementation details of our model are shown in Table TABREF31   For COCO / EMNLP2017, the generator is a LSTM unit BIBREF30 with 128 cells, and the discriminator is implemented based on BIBREF7   For WeiboDial, the generator is an encoder-decoder structure with attention mechanism, where both the encoder and the decoder consist of a two-layer GRU BIBREF31 with 128 cells  The discriminator is implemented based on BIBREF32   The language model used in the constrained sampling of ARAML is implemented in the same setting as the generators, and is pre-trained on the training set of each dataset  The codes and the datasets are available at https://github com/kepei1106/ARAML 
As for the details of the baselines, the generators of all the baselines except LeakGAN are the same as ours  Note that the generator of LeakGAN consists of a hierarchical LSTM unit, thus we followed the implementation in the original paper  In terms of the differences, the discriminators of GAN baselines are implemented based on the original papers  Other hyper-parameters of baselines including batch size, learning rate, and pre-training epochs, were set based on the original codes, because the convergence of baselines is sensitive to these hyper-parameters 
Language Generation on COCO and EMNLP2017 WMT
We adopted forward/reverse perplexity BIBREF33 and Self-BLEU BIBREF34 to evaluate the quality of generated texts  Forward perplexity (PPL-F) indicates the perplexity on the generated data provided by a language model trained on real data to measure the fluency of generated samples  Reverse perplexity (PPL-R) switches the roles of generated data and real data to reflect the discrepancy between the generated distribution and the data distribution  Self-BLEU (S-BLEU) regards each sentence in the generated collection as hypothesis and the others as reference to obtain BLEU scores, which evaluates the diversity of generated results 
Results are shown in Table TABREF33   LeakGAN performs best on forward perplexity because it can generate more fluent samples  As for reverse perplexity, our model ARAML beats other baselines, showing that our model can fit the data distribution better  Other GANs, particularly LeakGAN, obtain high reverse perplexity due to mode collapse BIBREF12 , thus they only capture limited fluent expressions, resulting in large discrepancy between the generated distribution and data distribution  ARAML also outperforms the baselines in terms of Self-BLEU, indicating that our model doesn't fall into mode collapse with the help of the MLE training objective and has the ability to generate more diverse sentences 
We also provide standard deviation of each metric in Table TABREF33 , reflecting the stability of each model's performance  Our model ARAML nearly achieves the smallest standard deviation in all the metrics, indicating that our framework outperforms policy gradient in the stability of adversarial training 
Dialogue Generation on WeiboDial
Dialogue evaluation is an open problem and existing works have found that automatic metrics have low correlation to human evaluation BIBREF35 , BIBREF36 , BIBREF37   Thus, we resorted to manual evaluation to assess the generation quality on WeiboDial  We randomly sampled 200 posts from the test set and collected the generated results from all the models  For each pair of responses (one from ARAML and the other from a baseline, given the same input post), five annotators were hired to label which response is better (i e  win, lose or tie) in terms of grammaticality (whether a response itself is grammatical and logical) and relevance (whether a response is appropriate and relevant to the post)  The two metrics were evaluated independently 
The evaluation results are shown in Table TABREF35   To measure the inter-annotator agreement, we calculated Fleiss' kappa BIBREF38 for each pair-wise comparison where results show moderate agreement ( INLINEFORM0 )  We also conducted sign test to check the significance of the differences 
As shown in Table TABREF35 , ARAML performs significantly better than other baselines in all the cases  This result indicates that the samples surrounding true responses provide stable rewards for the generator, and stable RAML training paradigm significantly enhances the performance in both metrics 
Further Analysis on Stability
To verify the training stability, we conducted experiments on COCO many times and chose the best 5 trials for SeqGAN, LeakGAN, IRL, MaliGAN and ARAML, respectively  Then, we presented the forward/reverse perplexity in the training process in Figure FIGREF38   We can see that our model with smaller standard deviation is more stable than other GAN baselines in both metrics  Although LeakGAN reaches the best forward perplexity, its standard deviation is extremely large and it performs badly in reverse perplexity, indicating that it generates limited expressions that are grammatical yet divergent from the data distribution 
Ablation Study
The temperature INLINEFORM0 controls the search space surrounding the real data as we analyze in Section UID13   To investigate its impact on the performance of our model, we fixed all the other hyper-parameters and test ARAML with different temperatures on COCO 
The experimental results are shown in Figure FIGREF41   We can see that as the temperature becomes larger, forward perplexity increases gradually while Self-BLEU decreases  As mentioned in Section UID13 , large temperatures encourage our generator to explore the samples that are distant from real data distribution, thus the diversity of generated results will be improved  However, these samples distant from the data distribution are more likely to be poor in fluency, leading to worse forward perplexity  Reverse perplexity is influenced by both generation quality and diversity, so the correlation between temperature and reverse perplexity is not intuitive  We can observe that the model with INLINEFORM0 reaches the best reverse perplexity 
We have mentioned two common sampling strategies in Section UID13 , i e  random sampling and constrained sampling  To analyze their impact, we keep all the model structures and hyper-parameters fixed and test ARAML with these two strategies on COCO 
Table TABREF45 shows the results  It's obvious that random sampling hurts the model performance except Self-BLEU-1, because it indeed allows low-quality samples available to the generator  Exploring these samples degrades the quality and diversity of generated results  Despite the worse performance on automatic metrics, random sampling doesn't affect the training stability of our framework  The standard deviation of ARAML-R is still smaller than other GAN baselines 
Case Study
Table TABREF47 presents the examples generated by the models on COCO  We can find that other baselines suffer from grammatical errors (e g  “in front of flying her kite" from MLE), repetitive expressions (e g  “A group of people" from IRL) and incoherent statements (e g  “A group of people sitting on a cell phone” from IRL)  By contrast, our model performs well in these sentences and has the ability to generate grammatical and coherent results 
Table TABREF48 shows the generated examples on WeiboDial  It's obvious that other baselines don't capture the topic word “late" in the post, thus generate irrelevant responses  ARAML can provide a response that is grammatical and closely relevant to the post 
Conclusion
We propose a novel adversarial training framework to deal with the instability problem of current GANs for text generation  To address the instability issue caused by policy gradient, we incorporate RAML into the advesarial training paradigm to make our generator acquire stable rewards  Experiments show that our model performs better than several state-of-the-art GAN baselines with lower training variance, yet producing better performance on three text generation tasks 
Acknowledgments
This work was supported by the National Science Foundation of China (Grant No  61936010/61876096) and the National Key R&D Program of China (Grant No  2018YFC0830200)  We would like to thank THUNUS NExT Joint-Lab for the support 

ground_truth: ARAM has achieved improvement over all baseline methods using reverese perplexity and  slef-BLEU metric.  The maximum reverse perplexity improvement 936,16 is gained for EMNLP2017  WMT  dataset and  48,44 for COCO dataset.

Compared to the baselines, ARAML does not do better in terms of perplexity on COCO and EMNLP 2017 WMT datasets, but it does by up to 0.27 Self-BLEU points on COCO and 0.35 Self-BLEU on EMNLP 2017 WMT. In terms of Grammaticality and Relevance, it scores better than the baselines on up to 75.5% and 73% of the cases respectively.