prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What metrics are used to establish that this makes chatbots more knowledgeable and better at learning and conversation? 
Context 1: Introduction
Chatbots such as dialog and question-answering systems have a long history in AI and natural language processing  Early such systems were mostly built using markup languages such as AIML, handcrafted conversation generation rules, and/or information retrieval techniques BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   Recent neural conversation models BIBREF4 , BIBREF5 , BIBREF6 are even able to perform open-ended conversations  However, since they do not use explicit knowledge bases and do not perform inference, they often suffer from generic and dull responses BIBREF5 , BIBREF7   More recently, BIBREF8 and BIBREF9 proposed to use knowledge bases (KBs) to help generate responses for knowledge-grounded conversation  However, one major weakness of all existing chat systems is that they do not explicitly or implicitly learn new knowledge in the conversation process  This seriously limits the scope of their applications  In contrast, we humans constantly learn new knowledge in our conversations  Even if some existing systems can use very large knowledge bases either harvested from a large data source such as the Web or built manually, these KBs still miss a large number of facts (knowledge) BIBREF10   It is thus important for a chatbot to continuously learn new knowledge in the conversation process to expand its KB and to improve its conversation ability 
In recent years, researchers have studied the problem of KB completion, i e , inferring new facts (knowledge) automatically from existing facts in a KB  KB completion (KBC) is defined as a binary classification problem: Given a query triple, ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ), we want to predict whether the source entity INLINEFORM3 and target entity INLINEFORM4 can be linked by the relation INLINEFORM5   However, existing approaches BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 solve this problem under the closed-world assumption, i e , INLINEFORM6 , INLINEFORM7 and INLINEFORM8 are all known to exist in the KB  This is a major weakness because it means that no new knowledge or facts may contain unknown entities or relations  Due to this limitation, KBC is clearly not sufficient for knowledge learning in conversations because in a conversation, the user can say anything, which may contain entities and relations that are not already in the KB 
In this paper, we remove this assumption of KBC, and allow all INLINEFORM0 , INLINEFORM1 and INLINEFORM2 to be unknown  We call the new problem open-world knowledge base completion (OKBC)  OKBC generalizes KBC  Below, we show that solving OKBC naturally provides the ground for knowledge learning and inference in conversations  In essence, we formulate an abstract problem of knowledge learning and inference in conversations as a well-defined OKBC problem in the interactive setting 
From the perspective of knowledge learning in conversations, essentially we can extract two key types of information, true facts and queries, from the user utterances  Queries are facts whose truth values need to be determined  Note that we do not study fact or relation extraction in this paper as there is an extensive work on the topic  (1) For a true fact, we will incorporate it into the KB  Here we need to make sure that it is not already in the KB, which involves relation resolution and entity linking  After a fact is added to the KB, we may predict that some related facts involving some existing relations in the KB may also be true (not logical implications as they can be automatically inferred)  For example, if the user says “Obama was born in USA,” the system may guess that (Obama, CitizenOf, USA) (meaning that Obama is a citizen of USA) could also be true based on the current KB  To verify this fact, it needs to solve a KBC problem by treating (Obama, CitizenOf, USA) as a query  This is a KBC problem because the fact (Obama, BornIn, USA) extracted from the original sentence has been added to the KB  Then Obama and USA are in the KB  If the KBC problem is solved, it learns a new fact (Obama, CitizenOf, USA) in addition to the extracted fact (Obama, BornIn, USA)  (2) For a query fact, e g , (Obama, BornIn, USA) extracted from the user question “Was Obama born in USA?” we need to solve the OKBC problem if any of “Obama, “BornIn”, or “USA" is not already in the KB 
We can see that OKBC is the core of a knowledge learning engine for conversation  Thus, in this paper, we focus on solving it  We assume that other tasks such as fact/relation extraction and resolution and guessing of related facts of an extracted fact are solved by other sub-systems 
We solve the OKBC problem by mimicking how humans acquire knowledge and perform reasoning in an interactive conversation  Whenever we encounter an unknown concept or relation while answering a query, we perform inference using our existing knowledge  If our knowledge does not allow us to draw a conclusion, we typically ask questions to others to acquire related knowledge and use it in inference  The process typically involves an inference strategy (a sequence of actions), which interleaves a sequence of processing and interactive actions  A processing action can be the selection of related facts, deriving inference chain, etc , that advances the inference process  An interactive action can be deciding what to ask, formulating a suitable question, etc , that enable us to interact  The process helps grow the knowledge over time and the gained knowledge enables us to communicate better in the future  We call this lifelong interactive learning and inference (LiLi)  Lifelong learning is reflected by the facts that the newly acquired facts are retained in the KB and used in inference for future queries, and that the accumulated knowledge in addition to the updated KB including past inference performances are leveraged to guide future interaction and learning  LiLi should have the following capabilities:
This setting is ideal for many NLP applications like dialog and question-answering systems that naturally provide the scope for human interaction and demand real-time inference 
LiLi starts with the closed-world KBC approach path-ranking (PR) BIBREF11 , BIBREF17 and extends KBC in a major way to open-world knowledge base completion (OKBC)  For a relation INLINEFORM0 , PR works by enumerating paths (except single-link path INLINEFORM1 ) between entity-pairs linked by INLINEFORM2 in the KB and use them as features to train a binary classifier to predict whether a query INLINEFORM3 should be in the KB  Here, a path between two entities is a sequence of relations linking them  In our work, we adopt the latest PR method, C-PR BIBREF16 and extend it to make it work in the open-world setting  C-PR enumerates paths by performing bidirectional random walks over the KB graph while leveraging the context of the source-target entity-pair  We also adopt and extend the compositional vector space model BIBREF20 , BIBREF21 with continual learning capability for prediction 
Given an OKBC query ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ) (e g , “(Obama, CitizenOf, USA), which means whether Obama a citizen of USA), LiLi interacts with the user (if needed) by dynamically formulating questions (see the interaction example in Figure 1, which will be further explained in §3) and leverages the interactively acquired knowledge (supporting facts (SFs) in the figure) for continued inference  To do so, LiLi formulates a query-specific inference strategy and executes it  We design LiLi in a Reinforcement Learning (RL) setting that performs sub-tasks like formulating and executing strategy, training a prediction model for inference, and knowledge retention for future use  To the best of our knowledge, our work is the first to address the OKBC problem and to propose an interactive learning mechanism to solve it in a continuous or lifelong manner  We empirically verify the effectiveness of LiLi on two standard real-world KBs: Freebase and WordNet  Experimental results show that LiLi is highly effective in terms of its predictive performance and strategy formulation ability 
Related Work
To the best of our knowledge, we are not aware of any knowledge learning system that can learn new knowledge in the conversation process  This section thus discusses other related work 
Among existing KB completion approaches, BIBREF20 extended the vector space model for zero-shot KB inference  However, the model cannot handle unknown entities and can only work on fixed set of unknown relations with known embeddings  Recently, BIBREF22 proposed a method using external text corpus to perform inference on unknown entities  However, the method cannot handle unknown relations  Thus, these methods are not suitable for our open-world setting  None of the existing KB inference methods perform interactive knowledge learning like LiLi  NELL BIBREF23 continuously updates its KB using facts extracted from the Web  Our task is very different as we do not do Web fact extraction (which is also useful)  We focus on user interactions in this paper  Our work is related to interactive language learning (ILL) BIBREF24 , BIBREF25 , but these are not about KB completion  The work in BIBREF26 allows a learner to ask questions in dialogue  However, this work used RL to learn about whether to ask the user or not  The “what to ask aspect" was manually designed by modeling synthetic tasks  LiLi formulates query-specific inference strategies which embed interaction behaviors  Also, no existing dialogue systems BIBREF4 , BIBREF27 , BIBREF28 , BIBREF29 , BIBREF30 employ lifelong learning to train prediction models by using information/knowledge retained in the past 
Our work is related to general lifelong learning in BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36   However, they learn only one type of tasks, e g , supervised, topic modeling or reinforcement learning (RL) tasks  None of them is suitable for our setting, which involves interleaving of RL, supervised and interactive learning  More details about lifelong learning can be found in the book BIBREF31  
Interactive Knowledge Learning (LiLi)
We design LiLi as a combination of two interconnected models: (1) a RL model that learns to formulate a query-specific inference strategy for performing the OKBC task, and (2) a lifelong prediction model to predict whether a triple should be in the KB, which is invoked by an action while executing the inference strategy and is learned for each relation as in C-PR  The framework improves its performance over time through user interaction and knowledge retention  Compared to the existing KB inference methods, LiLi overcomes the following three challenges for OKBC:
1  Mapping open-world to close-world  Being a closed-world method, C-PR cannot extract path features and learn a prediction model when any of INLINEFORM0 , INLINEFORM1 or INLINEFORM2 is unknown  LiLi solves this problem through interactive knowledge acquisition  If INLINEFORM3 is unknown, LiLi asks the user to provide a clue (an example of INLINEFORM4 )  And if INLINEFORM5 or INLINEFORM6 is unknown, LiLi asks the user to provide a link (relation) to connect the unknown entity with an existing entity (automatically selected) in the KB  We refer to such a query as a connecting link query (CLQ)  The acquired knowledge reduces OKBC to KBC and makes the inference task feasible 
2  Spareseness of KB  A main issue of all PR methods like C-PR is the connectivity of the KB graph  If there is no path connecting INLINEFORM0 and INLINEFORM1 in the graph, path enumeration of C-PR gets stuck and inference becomes infeasible  In such cases, LiLi uses a template relation (“@-?-@") as the missing link marker to connect entity-pairs and continues feature extraction  A path containing “@-?-@" is called an incomplete path  Thus, the extracted feature set contains both complete (no missing link) and incomplete paths  Next, LiLi selects an incomplete path from the feature set and asks the user to provide a link for path completion  We refer to such a query as missing link query (MLQ) 
3  Limitation in user knowledge  If the user is unable to respond to MLQs or CLQs, LiLi uses a guessing mechanism (discussed later) to fill the gap  This enables LiLi to continue its inference even if the user cannot answer a system question 
Components of LiLi
As lifelong learning needs to retain knowledge learned from past tasks and use it to help future learning BIBREF31 , LiLi uses a Knowledge Store (KS) for knowledge retention  KS has four components: (i) Knowledge Graph ( INLINEFORM0 ): INLINEFORM1 (the KB) is initialized with base KB triples (see §4) and gets updated over time with the acquired knowledge  (ii) Relation-Entity Matrix ( INLINEFORM2 ): INLINEFORM3 is a sparse matrix, with rows as relations and columns as entity-pairs and is used by the prediction model  Given a triple ( INLINEFORM4 , INLINEFORM5 , INLINEFORM6 ) INLINEFORM7 , we set INLINEFORM8 [ INLINEFORM9 , ( INLINEFORM10 , INLINEFORM11 )] = 1 indicating INLINEFORM12 occurs for pair ( INLINEFORM13 , INLINEFORM14 )  (iii) Task Experience Store ( INLINEFORM15 ): INLINEFORM16 stores the predictive performance of LiLi on past learned tasks in terms of Matthews correlation coefficient (MCC) that measures the quality of binary classification  So, for two tasks INLINEFORM17 and INLINEFORM18 (each relation is a task), if INLINEFORM19 [ INLINEFORM20 ] INLINEFORM21 INLINEFORM22 [ INLINEFORM23 ] [where INLINEFORM24 [ INLINEFORM25 ]=MCC( INLINEFORM26 )], we say C-PR has learned INLINEFORM27 well compared to INLINEFORM28   (iv) Incomplete Feature DB ( INLINEFORM29 ): INLINEFORM30 stores the frequency of an incomplete path INLINEFORM31 in the form of a tuple ( INLINEFORM32 , INLINEFORM33 , INLINEFORM34 ) and is used in formulating MLQs  INLINEFORM35 [( INLINEFORM36 , INLINEFORM37 , INLINEFORM38 )] = INLINEFORM39 implies LiLi has extracted incomplete path INLINEFORM40 INLINEFORM41 times involving entity-pair INLINEFORM42 [( INLINEFORM43 , INLINEFORM44 )] for query relation INLINEFORM45  
The RL model learns even after training whenever it encounters an unseen state (in testing) and thus, gets updated over time  KS is updated continuously over time as a result of the execution of LiLi and takes part in future learning  The prediction model uses lifelong learning (LL), where we transfer knowledge (parameter values) from the model for a past most similar task to help learn for the current task  Similar tasks are identified by factorizing INLINEFORM0 and computing a task similarity matrix INLINEFORM1   Besides LL, LiLi uses INLINEFORM2 to identify poorly learned past tasks and acquire more clues for them to improve its skillset over time 
LiLi also uses a stack, called Inference Stack ( INLINEFORM0 ) to hold query and its state information for RL  LiLi always processes stack top ( INLINEFORM1 [top])  The clues from the user get stored in INLINEFORM2 on top of the query during strategy execution and processed first  Thus, the prediction model for INLINEFORM3 is learned before performing inference on query, transforming OKBC to a KBC problem  Table 1 shows the parameters of LiLi used in the following sections 
Working of LiLi
Given an OKBC query ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ), we represent it as a data instance INLINEFORM3   INLINEFORM4 consists of INLINEFORM5 (the query triple), INLINEFORM6 (interaction limit set for INLINEFORM7 ), INLINEFORM8 (experience list storing the transition history of MDP for INLINEFORM9 in RL) and INLINEFORM10 (mode of INLINEFORM11 ) denoting if INLINEFORM12 is ` INLINEFORM13 ' (training), ` INLINEFORM14 ' (validation), ` INLINEFORM15 ' (evaluation) or ` INLINEFORM16 ' (clue) instance and INLINEFORM17 (feature set)  We denote INLINEFORM18 ( INLINEFORM19 ) as the set of all complete (incomplete) path features in INLINEFORM20   Given a data instance INLINEFORM21 , LiLi starts its initialization as follows: it sets the state as INLINEFORM22 (based on INLINEFORM23 , explained later), pushes the query tuple ( INLINEFORM24 , INLINEFORM25 ) into INLINEFORM26 and feeds INLINEFORM27 [top] to the RL-model for strategy formulation from INLINEFORM28  
Inference Strategy Formulation  We view solving the strategy formulation problem as learning to play an inference game, where the goal is to formulate a strategy that "makes the inference task possible"  Considering PR methods, inference is possible, iff (1) INLINEFORM0 becomes known to its KB (by acquiring clues when INLINEFORM1 is unknown) and (2) path features are extracted between INLINEFORM2 and INLINEFORM3 (which inturn requires INLINEFORM4 and INLINEFORM5 to be known to KB)  If these conditions are met at the end of an episode (when strategy formulation finishes for a given query) of the game, LiLi wins and thus, it trains the prediction model for INLINEFORM6 and uses it for inference 
LiLi's strategy formulation is modeled as a Markov Decision Process (MDP) with finite state ( INLINEFORM0 ) and action ( INLINEFORM1 ) spaces  A state INLINEFORM2 consists of 10 binary state variables (Table 2), each of which keeps track of results of an action INLINEFORM3 taken by LiLi and thus, records the progress in inference process made so far  INLINEFORM4 is the initial state with all state bits set as 0  If the data instance (query) is a clue [ INLINEFORM5 ], INLINEFORM6 [CLUE] is set as 1  INLINEFORM7 consists of 6 actions (Table 3)  INLINEFORM8 , INLINEFORM9 , INLINEFORM10 are processing actions and INLINEFORM11 , INLINEFORM12 , INLINEFORM13 are interactive actions  Whenever INLINEFORM14 is executed, the MDP reaches the terminal state  Given an action INLINEFORM15 in state INLINEFORM16 , if INLINEFORM17 is invalid in INLINEFORM21 or the objective of INLINEFORM22 is unsatisfied (* marked the condition in INLINEFORM23 ), RL receives a negative reward (empirically set); else receives a positive reward   We use Q-learning BIBREF38 with INLINEFORM24 -greedy strategy to learn the optimal policy for training the RL model  Note that, the inference strategy is independent of KB type and correctness of prediction  Thus, the RL-model is trained only once from scratch (reused thereafter for other KBs) and also, independently of the prediction model 
Sometimes the training dataset may not be enough to learn optimal policy for all INLINEFORM0   Thus, encountering an unseen state during test can make RL-model clueless about the action  Given a state INLINEFORM1 , whenever an invalid INLINEFORM2 is chosen, LiLi remains in INLINEFORM3   For INLINEFORM4 , LiLi remains in INLINEFORM5 untill INLINEFORM6 (see Table 1 for INLINEFORM7 )  So, if the state remains the same for ( INLINEFORM8 +1) times, it implies LiLi has encountered a fault (an unseen state)  RL-model instantly switches to the training mode and randomly explores INLINEFORM9 to learn the optimal action (fault-tolerant learning)  While exploring INLINEFORM10 , the model chooses INLINEFORM11 only when it has tried all other INLINEFORM12 to avoid abrupt end of episode 
Execution of Actions  At any given point in time, let ( INLINEFORM0 , INLINEFORM1 ) be the current INLINEFORM2 [top], INLINEFORM3 is the chosen action and the current version of KS components are INLINEFORM4 , INLINEFORM5 , INLINEFORM6 and INLINEFORM7   Then, if INLINEFORM8 is invalid in INLINEFORM9 , LiLi only updates INLINEFORM10 [top] with ( INLINEFORM11 , INLINEFORM12 ) and returns INLINEFORM13 [top] to RL-model  In this process, LiLi adds experience ( INLINEFORM14 , INLINEFORM15 , INLINEFORM16 , INLINEFORM17 ) in INLINEFORM18 and then, replaces INLINEFORM19 [top] with ( INLINEFORM20 , INLINEFORM21 )  If INLINEFORM22 is valid in INLINEFORM23 , LiLi first sets the next state INLINEFORM24 and performs a sequence of operations INLINEFORM25 based on INLINEFORM26 (discussed below)  Unless specified, in INLINEFORM27 , LiLi always monitors INLINEFORM28 and if INLINEFORM29 becomes 0, LiLi sets INLINEFORM30   Also, whenever LiLi asks the user a query, INLINEFORM31 is decremented by 1  Once INLINEFORM32 ends, LiLi updates INLINEFORM33 [top] with ( INLINEFORM34 , INLINEFORM35 ) and returns INLINEFORM36 [top] to RL-model for choosing the next action 
In INLINEFORM0 , LiLi searches INLINEFORM1 , INLINEFORM2 , INLINEFORM3 in INLINEFORM4 and sets appropriate bits in INLINEFORM5 (see Table 2)  If INLINEFORM6 was unknown before and is just added to INLINEFORM7 or is in the bottom INLINEFORM8 % (see Table 1 for INLINEFORM9 ) of INLINEFORM10 , LiLi randomly sets INLINEFORM14 with probability INLINEFORM15   If INLINEFORM16 is a clue and INLINEFORM17 , LiLi updates KS with triple INLINEFORM18 , where ( INLINEFORM19 , INLINEFORM20 , INLINEFORM21 ) and ( INLINEFORM22 , INLINEFORM23 , INLINEFORM24 ) gets added to INLINEFORM25 and INLINEFORM26 , INLINEFORM27 are set as 1 
In INLINEFORM0 , LiLi asks the user to provide a clue (+ve instance) for INLINEFORM1 and corrupts INLINEFORM2 and INLINEFORM3 of the clue once at a time, to generate -ve instances by sampling nodes from INLINEFORM4   These instances help in training prediction model for INLINEFORM5 while executing INLINEFORM6  
In INLINEFORM0 , LiLi selects an incomplete path INLINEFORM1 from INLINEFORM2 to formulate MLQ, such that INLINEFORM3 is most frequently observed for INLINEFORM4 and INLINEFORM5 is high, given by INLINEFORM6   Here, INLINEFORM7 denotes the contextual similarity BIBREF16 of entity-pair INLINEFORM8   If INLINEFORM9 is high, INLINEFORM10 is more likely to possess a relation between them and so, is a good candidate for formulating MLQ  When the user does not respond to MLQ (or CLQ in INLINEFORM11 ), the guessing mechanism is used, which works as follows: Since contextual similarity of entity-pairs is highly correlated with their class labels BIBREF16 , LiLi divides the similarity range [-1, 1] into three segments, using a low ( INLINEFORM12 ) and high ( INLINEFORM13 ) similarity threshold and replaces the missing link with INLINEFORM14 in INLINEFORM15 to make it complete as follows: If INLINEFORM16 , INLINEFORM17 = “@-LooselyRelatedTo-@"; else if INLINEFORM18 , INLINEFORM19 =“@-NotRelatedTo-@"; Otherwise, INLINEFORM20 =“@-RelatedTo-@" 
In INLINEFORM0 , LiLi asks CLQs for connecting unknown entities INLINEFORM1 and/or INLINEFORM2 with INLINEFORM3 by selecting the most contextually relevant node (wrt INLINEFORM4 , INLINEFORM5 ) from INLINEFORM6 , given by link INLINEFORM7   We adopt the contextual relevance idea in BIBREF16 which is computed using word embedding BIBREF39
In INLINEFORM0 , LiLi extracts path features INLINEFORM1 between ( INLINEFORM2 , INLINEFORM3 ) and updates INLINEFORM4 with incomplete features from INLINEFORM5   LiLi always trains the prediction model with complete features INLINEFORM6 and once INLINEFORM7 or INLINEFORM8 , LiLi stops asking MLQs  Thus, in both INLINEFORM9 and INLINEFORM10 , LiLi always monitors INLINEFORM11 to check for the said requirements and sets INLINEFORM12 to control interactions 
In INLINEFORM0 , if LiLi wins the episode, it adds INLINEFORM1 in one of data buffers INLINEFORM2 based on its mode INLINEFORM3   E g , if INLINEFORM4 or INLINEFORM5 , INLINEFORM6 is used for training and added to INLINEFORM7   Similarly validation buffer INLINEFORM8 and evaluation buffer INLINEFORM9 are populated  If INLINEFORM10 , LiLi invokes the prediction model for INLINEFORM11  
Lifelong Relation Prediction  Given a relation INLINEFORM0 , LiLi uses INLINEFORM1 and INLINEFORM2 (see INLINEFORM3 ) to train a prediction model (say, INLINEFORM4 ) with parameters INLINEFORM5   For a unknown INLINEFORM6 , the clue instances get stored in INLINEFORM7 and INLINEFORM8   Thus, LiLi populates INLINEFORM9 by taking 10% (see §4) of the instances from INLINEFORM10 and starts the training  For INLINEFORM11 , LiLi uses a LSTM BIBREF40 to compose the vector representation of each feature INLINEFORM12 as INLINEFORM13 and vector representation of INLINEFORM14 as INLINEFORM15   Next, LiLi computes the prediction value, INLINEFORM16 as sigmoid of the mean cosine similarity of all features and INLINEFORM17 , given by INLINEFORM18 ) and maximize the log-likelihood of INLINEFORM19 for training  Once INLINEFORM20 is trained, LiLi updates INLINEFORM21 [ INLINEFORM22 ] using INLINEFORM23   We also train an inverse model for INLINEFORM24 , INLINEFORM25 by reversing the path features in INLINEFORM26 and INLINEFORM27 which help in lifelong learning (discussed below)  Unlike BIBREF20 , BIBREF21 , while predicting the label for INLINEFORM28 , we compute a relation-specific prediction threshold INLINEFORM29 corresponding to INLINEFORM30 using INLINEFORM31 as: INLINEFORM32 and infer INLINEFORM33 as +ve if INLINEFORM34 and -ve otherwise  Here, INLINEFORM35 ( INLINEFORM36 ) is the mean prediction value for all +ve (-ve) examples in INLINEFORM37  
Models trained on a few examples (e g , clues acquired for unknown INLINEFORM0 ) with randomly initialized weights often perform poorly due to underfitting  Thus, we transfer knowledge (weights) from the past most similar (wrt INLINEFORM1 ) task in a lifelong learning manner BIBREF31   LiLi uses INLINEFORM2 to find the past most similar task for INLINEFORM3 as follows: LiLi computes trancated SVD of INLINEFORM4 as INLINEFORM5 and then, the similarity matrix INLINEFORM6   INLINEFORM7 provides the similarity between relations INLINEFORM8 and INLINEFORM9 in INLINEFORM10   Thus, LiLi chooses a source relation INLINEFORM11 to transfer weights  Here, INLINEFORM12 is the set of all INLINEFORM13 and INLINEFORM14 for which LiLi has already learned a prediction model  Now, if INLINEFORM15 or INLINEFORM16 , LiLi randomly initializes the weights INLINEFORM17 for INLINEFORM18 and proceeds with the training  Otherwise, LiLi uses INLINEFORM19 as initial weights and fine-tunes INLINEFORM20 with a low learning rate 
A Running Example  Considering the example shown in Figure 1, LiLi works as follows: first, LiLi executes INLINEFORM0 and detects that the source entity “Obama" and query relation “CitizenOf" are unknown  Thus, LiLi executes INLINEFORM1 to acquire clue (SF1) for “CitizenOf" and pushes the clue (+ve example) and two generated -ve examples into INLINEFORM2   Once the clues are processed and a prediction model is trained for “CitizenOf" by formulating separate strategies for them, LiLi becomes aware of “CitizenOf"  Now, as the clues have already been popped from INLINEFORM3 , the query becomes INLINEFORM4 and the strategy formulation process for the query resumes  Next, LiLi asks user to provide a connecting link for “Obama" by performing INLINEFORM5   Now, the query entities and relation being known, LiLi enumerates paths between “Obama" and “USA" by performing INLINEFORM6   Let an extracted path be “ INLINEFORM7 " with missing link between ( INLINEFORM8 , INLINEFORM9 )  LiLi asks the user to fill the link by performing INLINEFORM10 and then, extracts the complete feature “ INLINEFORM11 "  The feature set is then fed to the prediction model and inference is made as a result of INLINEFORM12   Thus, the formulated inference strategy is: “ INLINEFORM13 " 
Experiments
We now evaluate LiLi in terms of its predictive performance and strategy formulation abilities 
Data: We use two standard datasets (see Table 4): (1) Freebase FB15k, and (2) WordNet INLINEFORM0   Using each dataset, we build a fairly large graph and use it as the original KB ( INLINEFORM1 ) for evaluation  We also augment INLINEFORM2 with inverse triples ( INLINEFORM3 , INLINEFORM4 , INLINEFORM5 ) for each ( INLINEFORM6 , INLINEFORM7 , INLINEFORM8 ) following existing KBC methods 
Parameter Settings  Unless specified, the empirically set parameters (see Table 1) of LiLi are: INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 , INLINEFORM8 , INLINEFORM9 , INLINEFORM10   For training RL-model with INLINEFORM11 -greedy strategy, we use INLINEFORM12 , INLINEFORM13 , pre-training steps=50000  We used Keras deep learning library to implement and train the prediction model  We set batch-size as 128, max  training epoch as 150, dropout as 0 2, hidden units and embedding size as 300 and learning rate as 5e-3 which is reduced gradually on plateau with factor 0 5 and patience 5  Adam optimizer and early stopping were used in training  We also shuffle INLINEFORM14 in each epoch and adjust class weights inversely proportional to class frequencies in INLINEFORM15  
Labeled Dataset Generation and Simulated User Creation  We create a simulated user for each KB to evaluate LiLi  We create the labeled datasets, the simulated user’s knowledge base ( INLINEFORM0 ), and the base KB ( INLINEFORM1 ) from INLINEFORM2   INLINEFORM3 used as the initial KB graph ( INLINEFORM4 ) of LiLi 
We followed BIBREF16 for labeled dataset generation  For Freebase, we found 86 relations with INLINEFORM0 triples and randomly selected 50 from various domains  We randomly shuffle the list of 50 relations, select 25% of them as unknown relations and consider the rest (75%) as known relations  For each known relation INLINEFORM1 , we randomly shuffle the list of distinct triples for INLINEFORM2 , choose 1000 triples and split them into 60% training, 10% validation and 20% test  Rest 10% along with the leftover (not included in the list of 1000) triples are added to INLINEFORM3   For each unknown relation INLINEFORM4 , we remove all triples of INLINEFORM5 from INLINEFORM6 and add them to INLINEFORM7   In this process, we also randomly choose 20% triples as test instances for unknown INLINEFORM8 which are excluded from INLINEFORM9   Note that, now INLINEFORM10 has at least 10% of chosen triples for each INLINEFORM11 (known and unknown) and so, user is always able to provide clues for both cases  For each labeled dataset, we randomly choose 10% of the entities present in dataset triples, remove triples involving those entities from INLINEFORM12 and add to INLINEFORM13   At this point, INLINEFORM14 gets reduced to INLINEFORM15 and is used as INLINEFORM16 for LiLi  The dataset stats in Table 4 shows that the base KB (60% triples of INLINEFORM17 ) is highly sparse (compared to original KB) which makes the inference task much harder  WordNet dataset being small, we select all 18 relations for evaluation and create labeled dataset, INLINEFORM18 and INLINEFORM19 following Freebase  Although the user may provide clues 100% of the time, it often cannot respond to MLQs and CLQs (due to lack of required triples/facts)  Thus, we further enrich INLINEFORM20 with external KB triples 
Given a relation INLINEFORM0 and an observed triple ( INLINEFORM1 , INLINEFORM2 , INLINEFORM3 ) in training or testing, the pair ( INLINEFORM4 , INLINEFORM5 ) is regarded as a +ve instance for INLINEFORM6   Following BIBREF18 , for each +ve instance ( INLINEFORM7 , INLINEFORM8 ), we generate two negative ones, one by randomly corrupting the source INLINEFORM9 , and the other by corrupting the target INLINEFORM10   Note that, the test triples are not in INLINEFORM11 or INLINEFORM12 and none of the -ve instances overlap with the +ve ones 
Baselines  As none of the existing KBC methods can solve the OKBC problem, we choose various versions of LiLi as baselines 
Single: Version of LiLi where we train a single prediction model INLINEFORM0 for all test relations 
Sep: We do not transfer (past learned) weights for initializing INLINEFORM0 , i e , we disable LL 
F-th): Here, we use a fixed prediction threshold 0 5 instead of relation-specific threshold INLINEFORM0  
BG: The missing or connecting links (when the user does not respond) are filled with “@-RelatedTo-@" blindly, no guessing mechanism 
w/o PTS: LiLi does not ask for additional clues via past task selection for skillset improvement 
Evaluation Metrics  To evaluate the strategy formulation ability, we introduce a measure called Coverage( INLINEFORM0 ), defined as the fraction of total query data instances, for which LiLi has successfully formulated strategies that lead to winning  If LiLi wins on all episodes for a given dataset, INLINEFORM1 is 1 0  To evaluate the predictive performance, we use Avg  MCC and avg  +ve F1 score 
Results and Analysis
Evaluation-I: Strategy Formulation Ability  Table 5 shows the list of inference strategies formulated by LiLi for various INLINEFORM0 and INLINEFORM1 , which control the strategy formulation of LiLi  When INLINEFORM2 , LiLi cannot interact with user and works like a closed-world method  Thus, INLINEFORM3 drops significantly (0 47)  When INLINEFORM4 , i e  with only one interaction per query, LiLi acquires knowledge well for instances where either of the entities or relation is unknown  However, as one unknown entity may appear in multiple test triples, once the entity becomes known, LiLi doesn’t need to ask for it again and can perform inference on future triples causing significant increase in INLINEFORM5 (0 97)  When INLINEFORM6 , LiLi is able to perform inference on all instances and INLINEFORM7 becomes 1  For INLINEFORM8 , LiLi uses INLINEFORM9 only once (as only one MLQ satisfies INLINEFORM10 ) compared to INLINEFORM11   In summary, LiLi’s RL-model can effectively formulate query-specific inference strategies (based on specified parameter values)  Evaluation-II: Predictive Performance  Table 6 shows the comparative performance of LiLi with baselines  To judge the overall improvements, we performed paired t-test considering +ve F1 scores on each relation as paired data  Considering both KBs and all relation types, LiLi outperforms Sep with INLINEFORM12   If we set INLINEFORM13 (training with very few clues), LiLi outperforms Sep with INLINEFORM14 on Freebase considering MCC  Thus, the lifelong learning mechanism is effective in transferring helpful knowledge  Single model performs better than Sep for unknown relations due to the sharing of knowledge (weights) across tasks  However, for known relations, performance drops because, as a new relation arrives to the system, old weights get corrupted and catastrophic forgetting occurs  For unknown relations, as the relations are evaluated just after training, there is no chance for catastrophic forgetting  The performance improvement ( INLINEFORM15 ) of LiLi over F-th on Freebase signifies that the relation-specific threshold INLINEFORM16 works better than fixed threshold 0 5 because, if all prediction values for test instances lie above (or below) 0 5, F-th predicts all instances as +ve (-ve) which degrades its performance  Due to the utilization of contextual similarity (highly correlated with class labels) of entity-pairs, LiLi’s guessing mechanism works better ( INLINEFORM17 ) than blind guessing (BG)  The past task selection mechanism of LiLi also improves its performance over w/o PTS, as it acquires more clues during testing for poorly performed tasks (evaluated on validation set)  For Freebase, due to a large number of past tasks [9 (25% of 38)], the performance difference is more significant ( INLINEFORM18 )  For WordNet, the number is relatively small [3 (25% of 14)] and hence, the difference is not significant 
Evaluation-III: User Interaction vs  Performance  Table 7 shows the results of LiLi by varying clue acquisition rate ( INLINEFORM0 )  We use Freebase for tuning INLINEFORM1 due to its higher number of unknown test relations compared to WordNet  LiLi’s performance improves significantly as it acquires more clues from the user  The results on INLINEFORM2 outperforms ( INLINEFORM3 ) that on INLINEFORM4   Table 8 shows the results of LiLi on user responses to MLQ’s and CLQ’s  Answering MLQ’s and CLQ’s is very hard for simulated users (unlike crowd-sourcing) as often INLINEFORM5 lacks the required triple  Thus, we attempt to analyze how the performance is effected if the user does not respond at all  The results show a clear trend in overall performance improvement when the user responds  However, the improvement is not significant as the simulated user’s query satisfaction rate (1% MLQs and 10% CLQs) is very small  But, the analysis shows the effectiveness of LiLi’s guessing mechanism and continual learning ability that help in achieving avg  +ve F1 of 0 57 and 0 62 on FB and WN respectively with minimal participation of the user 
Conclusion
In this paper, we are interested in building a generic engine for continuous knowledge learning in human-machine conversations  We first showed that the problem underlying the engine can be formulated as an open-world knowledge base completion (OKBC) problem  We then proposed an lifelong interactive learning and inference (LiLi) approach to solving the OKBC problem  OKBC is a generalization of KBC  LiLi solves the OKBC problem by first formulating a query-specific inference strategy using RL and then executing it to solve the problem by interacting with the user in a lifelong learning manner  Experimental results showed the effectiveness of LiLi in terms of both predictive quality and strategy formulation ability  We believe that a system with the LiLi approach can serve as a knowledge learning engine for conversations  Our future work will improve LiLi to make more accurate 
Acknowledgments
This work was supported in part by National Science Foundation (NSF) under grant no  IIS-1407927 and IIS-1650900, and a gift from Huawei Technologies Co Ltd 
Context 2: Introduction
Part-of-speech tagging is now a classic task in natural language processing, for which many systems have been developed or adapted for a large variety of languages  Its aim is to associate each “word” with a morphosyntactic tag, whose granularity can range from a simple morphosyntactic category, or part-of-speech (hereafter PoS), to finer categories enriched with morphological features (gender, number, case, tense, mood, etc ) 
The use of machine learning algorithms trained on manually annotated corpora has long become the standard way to develop PoS taggers  A large variety of algorithms have been used, such as (in approximative chronological order) bigram and trigram hidden Markov models BIBREF0 , BIBREF1 , BIBREF2 , decision trees BIBREF3 , BIBREF4 , maximum entropy Markov models (MEMMs) BIBREF5 and Conditional Random Fields (CRFs) BIBREF6 , BIBREF7   With such machine learning algorithms, it is possible to build PoS taggers for any language, provided adequate training data is available 
As a complement to annotated corpora, it has previously been shown that external lexicons are valuable sources of information, in particular morphosyntactic lexicons, which provide a large inventory of (word, PoS) pairs  Such lexical information can be used in the form of constraints at tagging time BIBREF8 , BIBREF9 or during the training process as additional features combined with standard features extracted from the training corpus BIBREF10 , BIBREF11 , BIBREF12  
In recent years, a different approach to modelling lexical information and integrating it into natural language processing systems has emerged, namely the use of vector representations for words or word sequences BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18   Such representations, which are generally extracted from large amounts of raw text, have proved very useful for numerous tasks including PoS tagging, in particular when used in recurrent neural networks (RNNs) and more specifically in mono- or bi-directional, word-level and/or character-level long short-term memory networks (LSTMs) BIBREF19 , BIBREF16 , BIBREF17 , BIBREF20  
Both approaches to representing lexical properties and to integrating them into a PoS tagger improve tagging results  Yet they rely on resources of different natures  The main advantage of word vectors is that they are built in an unsupervised way, only requiring large amounts of raw textual data  They also encode finer-grained information than usual morphosyntactic lexicons, most of which do not include any quantitative data, not even simple frequency information  Conversely, lexical resources often provide information about scarcely attested words, for which corpus-based approaches such as word vector representations are of limited relevance  Moreover, morphological or morphosyntactic lexicons already exist for a number of languages, including less-resourced langauges for which it might be difficult to obtain the large amounts of raw data necessary to extract word vector representations 
Our main goal is therefore to compare the respective impact of external lexicons and word vector representations on the accuracy of PoS models  This question has already been investigated for 6 languages by BIBREF18 using the state-of-the-art CRF-based tagging system MarMoT  The authors found that their best-performing word-vector-based PoS tagging models outperform their models that rely on morphosyntactic resources (lexicons or morphological analysers)  In this paper, we report on larger comparison, carried out in a larger multilingual setting and comparing different tagging models  Using different 16 datasets, we compare the performances of two feature-based models enriched with external lexicons and of two LSTM-based models enriched with word vector representations  A secondary goal of our work is to compare the relative improvements linked to the use of external lexical information in the two feature-based models, which use different models (MEMM vs  CRF) and feature sets 
More specifically, our starting point is the MElt system BIBREF12 , an MEMM tagging system  We first briefly describe this system and the way we adapted it by integrating our own set of corpus-based and lexical features  We then introduce the tagging models we have trained for 16 different languages using our adapted version of MElt  These models are trained on the Universal Dependencies (v1 2) corpus set BIBREF21 , complemented by morphosyntactic lexicons  We compare the accuracy of our models with the scores obtained by the CRF-based system MarMoT BIBREF22 , BIBREF18 , retrained on the same corpora and the same external morphosyntactic lexicons  We also compare our results to those obtained by the best bidirectional LSTM models described by BIBREF20 , which both make use of Polyglot word vector representations published by BIBREF23   We will show that an optimised enrichment of feature-based models with morphosyntactic lexicon results in significant accuracy gains  The macro-averaged accuracy of our enriched MElt models is above that of enriched MarMoT models and virtually identical to that of LSTMs enriched with word vector representations  More precisely, per-language results indicate that lexicons provide more useful information for languages with a high lexical variability (such as morphologically rich languages), whereas word vectors are more informative for languages with a lower lexical variability (such as English) 
MElt
MElt BIBREF12 is a tagging system based on maximum entropy Markov models (MEMM) BIBREF5 , a class of discriminative models that are suitable for sequence labelling BIBREF5   The basic set of features used by MElt is given in BIBREF12   It is a superset of the feature sets used by BIBREF5 and BIBREF24 and includes both local standard features (for example the current word itself and its prefixes and suffixes of length 1 to 4) and contextual standard features (for example the tag just assigned to the preceding word)  In particular, with respect to Ratnaparkhi's feature set, MElt's basic feature set lifts the restriction that local standard features used to analyse the internal composition of the current word should only apply to rare words 
One of the advantages of feature-based models such as MEMMs and CRFs is that complementary information can be easily added in the form of additional features  This was investigated for instance by BIBREF25 , whose best-performing model for PoS tagging dialogues was obtained with a version of MElt extended with dialogue-specific features  Yet the motivation of MElt's developers was first and foremost to investigate the best way to integrate lexical information extracted from large-scale morphosyntactic lexical resources into their models, on top of the training data BIBREF12   They showed that performances are better when this external lexical information is integrated in the form of additional lexical features than when the external lexicon is used as constraints at tagging time  These lexical features can also be divided into local lexical features (for example the list of possible tags known to the external lexicon for the current word) and contextual lexical features (for example the list of possible tags known to the external lexicon for surrounding words)  In particular, lexical contextual features provide a means to model the right context of the current word, made of words that have not yet been tagged by the system but for which the lexicon often provides a list of possible tags  Moreover, tagging accuracy for out-of-vocabulary (OOV) words is improved, as a result of the fact that words unknown to the training corpus might be known to the external lexicon 
Despite a few experiments published with MElt on languages other than French BIBREF12 , BIBREF40 , BIBREF41 , the original feature set used by MElt (standard and lexical features) was designed and tested mostly on this language, by building and evaluating tagging models on a variant of the French TreeBank  Since our goal was to carry out experiments in a multilingual setting, we have decided to design our own set of features, using the standard MElt features as a starting point  With respect to the original MElt feature set, we have added new ones, such as prefixes and suffixes of the following word, as well as a hybrid contextual feature obtained by concatenating the tag predicted for the preceding word and the tag(s) provided by the external lexicon for the following word 
In order to select the best performing feature set, we carried out a series of experiments using the multilingual dataset provided during the SPMRL parsing shared task BIBREF42   This included discarding useless or harmful features and selecting the maximal length of the prefixes and suffixes to be used as features, both for the current word and for the following word 
We incorporated in MElt the best performing feature set, described in Table TABREF1   All models discussed in this paper are based on this feature set 
Corpora
We carried out our experiments on the Universal Dependencies v1 2 treebanks BIBREF21 , hereafter UD1 2, from which morphosyntactically annotated corpora can be trivially extracted  All UD1 2 corpora use a common tag set, the 17 universal PoS tags, which is an extension of the tagset proposed by BIBREF43  
As our goal is to study the impact of lexical information for PoS tagging, we have restricted our experiments to UD1 2 corpora that cover languages for which we have morphosyntactic lexicons at our disposal, and for which BIBREF20 provide results  We considered UD1 2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish  Although this language list contains only one non-Indo-European (Indonesian), four major Indo-European sub-families are represented (Germanic, Romance, Slavic, Indo-Iranian)  Overall, the 16 languages considered in our experiments are typologically, morphologically and syntactically fairly diverse 
Lexicons
We generate our external lexicons using the set of source lexicons listed in Table TABREF3   Since external lexical information is exploited via features, there is no need for the external lexicons and the annotated corpora to use the same PoS inventory  Therefore, for each language, we simply extracted from the corresponding lexicon the PoS of each word based on its morphological tags, by removing all information provided except for its coarsest-level category  We also added entries for punctuations when the source lexicons did not contain any 
We also performed experiments in which we retained the full original tags provided by the lexicons, with all morphological features included  On average, results were slightly better than those presented in the paper, although not statistically significantly  Moreover, the granularity of tag inventories in the lexicons is diverse, which makes it difficult to draw general conclusions about results based on full tags  This is why we only report results based on (coarse) PoS extracted from the original lexicons 
Baseline models
In order to assess the respective contributions of external lexicons and word vector representations, we first compared the results of the three above-mentioned systems when trained without such additional lexical information  Table TABREF11 provides the results of MElt and MarMoT retrained on UD1 2 corpora, together with the results publised on the same corpora by BIBREF20 , using their best model not enhanced by external word vector representations —i e  the model they call INLINEFORM0 , which is a bidirectional LSTM that combines both word and character embeddings 
These results show that Plank et al 's (2016) bi-LSTM performs extremely well, surpassed by MarMoT on only 3 out of 16 datasets (Czech, French and Italian), and by MElt only once (Indonesian) 
Models enriched with external lexical information
Table TABREF13 provides the results of four systems enriched with lexical information  The feature-based systems MElt and MarMoT, respectively based on MEMMs and CRFs, are extended with the lexical information provided by our morphosyntactic lexicons  This extension takes the form of additional features, as described in Section SECREF2 for MElt  The results reported by BIBREF20 for their bidirectional LSTM when initialised with Polyglot embeddings trained on full wikipedias are also included, together with their new system FREQBIN, also initialised with Polyglot embeddings  FREQBIN trains bi-LSTMs to predict for each input word both a PoS and a label that represents its log frequency in the training data  As they word it, “the idea behind this model is to make the representation predictive for frequency, which encourages the model not to share representations between common and rare words, thus benefiting the handling of rare tokens ”
The results, which are also displayed in Figures FIGREF14 and FIGREF15 , show that all systems reach very similar results on average, although discrepancies can be observed from one dataset to another, on which we shall comment shortly  The best performing system in terms of macro-average is MElt (96 60%)  Both bi-LSTM systems reach the same score (96 58%), the difference with MElt's results being non significant, whereas MarMoT is only 0 14% behind (96 46%)  Given the better baseline scores of the neural approaches, these results show that the benefit of using external lexicons in the feature-based models MElt and MarMoT are much higher than those using Polyglot word vector representations as initialisations for bi-LSTMs 
Yet these very similar overall results reflect a different picture when focusing on OOV tagging accuracy  The best models for OOV tagging accuracy are, by far, FREQBIN models, which are beaten by MarMoT and by MElt only once each (on English and Danish respectively)  The comparison on OOV tagging between MElt and MarMoT shows that MElt performs better on average than MarMoT, despite the fact that MarMoT's baseline results were better than those reached by MElt  This shows that the information provided by external morphosyntactic lexicons is better exploited by MElt's lexical features than by those used by MarMoT  On the other hand, the comparison of both bi-LSTM-based approaches confirm that the FREQBIN models is better by over 10% absolute on OOV tagging accuracy (94 28% vs  83 59%), with 65% lower error rate 
One of the important differences between the lexical information provided by an external lexicon and word vectors built from raw corpora, apart from the very nature of the lexical information provided, is the coverage and accuracy of this lexical information on rare words  All words in a morphosyntactic lexicon are associated with information of a same granularity and quality, which is not the case with word representations such as provided by Polyglot  Models that take advantage of external lexicons should therefore perform comparatively better on datasets containing a higher proportion of rarer words, provided the lexicons' coverage is high  In order to confirm this intuition, we have used a lexical richness metric based on the type/token ratio  Since this ratio is well-known for being sensitive to corpus length, we normalised it by computing it over the 60,000 first tokens of each training set  When this normalised type/token ratio is plotted against the difference between the results of MElt and both bi-LSTM-based models, the expected correlation is clearly visible (see Figure FIGREF16 )  This explains why MElt obtains better results on the morphologically richer Slavic datasets (average normalised type/token ratio: 0 28, average accuracy difference: 0 32 compared to both bi-LSTM+Polyglot and FREQBIN+Polyglot) and, at the other end of the spectrum, significantly worse results on the English dataset (normalised type/token ratio: 0 15, average accuracy difference: -0 56 compared to bi-LSTM+Polyglot, -0 57 compared to FREQBIN+Polyglot) 
Conclusion
Two main conclusions can be drawn from our comparative results  First, feature-based tagging models adequately enriched with external morphosyntactic lexicons perform, on average, as well as bi-LSTMs enriched with word embeddings  Per-language results show that the best accuracy levels are reached by feature-based models, and in particular by our improved version of the MEMM-based system MElt, on datasets with high lexical variability (in short, for morphologically rich languages), whereas neural-based results perform better on datatsets with lower lexical variability (e g  for English) 
We have only compared the contribution of morphosyntactic lexicons to feature-based models (MEMMs, CRFs) and that of word vector representations to bi-LSTM-based models as reported by BIBREF20   As mentioned above, work on the contribution of word vector representations to feature-based approaches has been carried out by BIBREF18   However, the exploitation of existing morphosyntactic or morphological lexicons in neural models is a less studied question  Improvements over the state of the art might be achieved by integrating lexical information both from an external lexicon and from word vector representations into tagging models 
In that regard, further work will be required to understand which class of models perform the best  An option would be to integrate feature-based models such as a CRF with an LSTM-based layer, following recent proposals such as the one proposed by BIBREF45 for named entity recognition 
Context 3: INTRODUCTION
The idea of language identification is to classify a given audio signal into a particular class using a classification algorithm  Commonly language identification task was done using i-vector systems [1]  A very well known approach for language identification proposed by N  Dahek et al  [1] uses the GMM-UBM model to obtain utterance level features called i-vectors  Recent advances in deep learning [15,16] have helped to improve the language identification task using many different neural network architectures which can be trained efficiently using GPUs for large scale datasets  These neural networks can be configured in various ways to obtain better accuracy for language identification task  Early work on using Deep learning for language Identification was published by Pavel Matejka et al  [2], where they used stacked bottleneck features extracted from deep neural networks for language identification task and showed that the bottleneck features learned by Deep neural networks are better than simple MFCC or PLP features  Later the work by I  Lopez-Moreno et al  [3] from Google showed how to use Deep neural networks to directly map the sequence of MFCC frames into its language class so that we can apply language identification at the frame level  Speech signals will have both spatial and temporal information, but simple DNNs are not able to capture temporal information  Work done by J  Gonzalez-Dominguez et al  [4] by Google developed an LSTM based language identification model which improves the accuracy over the DNN based models  Work done by Alicia et al  [5] used CNNs to improve upon i-vector [1] and other previously developed systems  The work done by Daniel Garcia-Romero et al  [6] has used a combination of Acoustic model trained for speech recognition with Time-delay neural networks where they train the TDNN model by feeding the stacked bottleneck features from acoustic model to predict the language labels at the frame level  Recently X-vectors [7] is proposed for speaker identification task and are shown to outperform all the previous state of the art speaker identification algorithms and are also used for language identification by David Snyder et al  [8] 
In this paper, we explore multiple pooling strategies for language identification task  Mainly we propose Ghost-VLAD based pooling method for language identification  Inspired by the recent work by W  Xie et al  [9] and Y  Zhong et al  [10], we use Ghost-VLAD to improve the accuracy of language identification task for Indian languages  We explore multiple pooling strategies including NetVLAD pooling [11], Average pooling and Statistics pooling( as proposed in X-vectors [7]) and show that Ghost-VLAD pooling is the best pooling strategy for language identification  Our model obtains the best accuracy of 98 24%, and it outperforms all the other previously proposed pooling methods  We conduct all our experiments on 635hrs of audio data for 7 Indian languages collected from $\textbf {All India Radio}$ news channel  The paper is organized as follows  In section 2, we explain the proposed pooling method for language identification  In section 3, we explain our dataset  In section 4, we describe the experiments, and in section 5, we describe the results 
POOLING STRATEGIES
In any language identification model, we want to obtain utterance level representation which has very good language discriminative features  These representations should be compact and should be easily separable by a linear classifier  The idea of any pooling strategy is to pool the frame-level representations into a single utterance level representation  Previous works by [7] have used simple mean and standard deviation aggregation to pool the frame-level features from the top layer of the neural network to obtain the utterance level features  Recently [9] used VLAD based pooling strategy for speaker identification which is inspired from [10] proposed for face recognition  The NetVLAD [11] and Ghost-VLAD [10] methods are proposed for Place recognition and face recognition, respectively, and in both cases, they try to aggregate the local descriptors into global features  In our case, the local descriptors are features extracted from ResNet [15], and the global utterance level feature is obtained by using GhostVLAD pooling  In this section, we explain different pooling methods, including NetVLAD, Ghost-VLAD, Statistic pooling, and Average pooling 
POOLING STRATEGIES ::: NetVLAD pooling
The NetVLAD pooling strategy was initially developed for place recognition by R  Arandjelovic et al  [11]  The NetVLAD is an extension to VLAD [18] approach where they were able to replace the hard assignment based clustering with soft assignment based clustering so that it can be trained with neural network in an end to end fashion  In our case, we use the NetVLAD layer to map N local features of dimension D into a fixed dimensional vector, as shown in Figure 1 (Left side) 
The model takes spectrogram as an input and feeds into CNN based ResNet architecture  The ResNet is used to map the spectrogram into 3D feature map of dimension HxWxD  We convert this 3D feature map into 2D by unfolding H and W dimensions, creating a NxD dimensional feature map, where N=HxW  The NetVLAD layer is kept on top of the feature extraction layer of ResNet, as shown in Figure 1  The NetVLAD now takes N features vectors of dimension D and computes a matrix V of dimension KxD, where K is the number clusters in the NetVLAD layer, and D is the dimension of the feature vector  The matrix V is computed as follows 
Where $w_k$,$b_k$ and $c_k$ are trainable parameters for the cluster $k$ and V(j,k) represents a point in the V matrix for (j,k)th location  The matrix is constructed using the equation (1) where the first term corresponds to the soft assignment of the input $x_i$ to the cluster $c_k$, whereas the second term corresponds to the residual term which tells how far the input descriptor $x_i$ is from the cluster center $c_k$ 
POOLING STRATEGIES ::: GhostVLAD pooling
GhostVLAD is an extension of the NetVLAD approach, which we discussed in the previous section  The GhostVLAD model was proposed for face recognition by Y  Zhong [10]  GhostVLAD works exactly similar to NetVLAD except it adds Ghost clusters along with the NetVLAD clusters  So, now we will have a K+G number of clusters instead of K clusters  Where G is the number of ghost clusters, we want to add (typically 2-4)  The Ghost clusters are added to map any noisy or irrelevant content into ghost clusters and are not included during the feature aggregation stage, as shown in Figure 1 (Right side)  Which means that we compute the matrix V for both normal cluster K and ghost clusters G, but we will not include the vectors belongs to ghost cluster from V during concatenation of the features  Due to which, during feature aggregation stage the contribution of the noisy and unwanted features to normal VLAD clusters are assigned less weights while Ghost clusters absorb most of the weight  We illustrate this in Figure 1(Right Side), where the ghost clusters are shown in red color  We use Ghost clusters when we are computing the V matrix, but they are excluded during the concatenation stage  These concatenated features are fed into the projection layer, followed by softmax to predict the language label 
POOLING STRATEGIES ::: Statistic and average pooling
In statistic pooling, we compute the first and second order statistics of the local features from the top layer of the ResNet model  The 3-D feature map is unfolded to create N features of D dimensions, and then we compute the mean and standard deviation of all these N vectors and get two D dimensional vectors, one for mean and the other for standard deviation  We then concatenate these 2 features and feed it to the projection layer for predicting the language label 
In the Average pooling layer, we compute only the first-order statistics (mean) of the local features from the top layer of the CNN model  The feature map from the top layer of CNN is unfolded to create N features of D dimensions, and then we compute the mean of all these N vectors and get D dimensional representation  We then feed this feature to the projection layer followed by softmax for predicting the language label 
DATASET
In this section, we describe our dataset collection process  We collected and curated around 635Hrs of audio data for 7 Indian languages, namely Kannada, Hindi, Telugu, Malayalam, Bengali, and English  We collected the data from the All India Radio news channel where an actor will be reading news for about 5-10 mins  To cover many speakers for the dataset, we crawled data from 2010 to 2019  Since the audio is very long to train any deep neural network directly, we segment the audio clips into smaller chunks using Voice activity detector  Since the audio clips will have music embedded during the news, we use Inhouse music detection model to remove the music segments from the dataset to make the dataset clean and our dataset contains 635Hrs of clean audio which is divided into 520Hrs of training data containing 165K utterances and 115Hrs of testing data containing 35K utterances  The amount of audio data for training and testing for each of the language is shown in the table bellow 
EXPERIMENTS
In this section, we describe the feature extraction process and network architecture in detail  We use spectral features of 256 dimensions computed using 512 point FFT for every frame, and we add an energy feature for every frame giving us total 257 features for every frame  We use a window size of 25ms and frame shift of 10ms during feature computation  We crop random 5sec audio data from each utterance during training which results in a spectrogram of size 257x500 (features x number of features)  We use these spectrograms as input to our CNN model during training  During testing, we compute the prediction score irrespective of the audio length 
For the network architecture, we use ResNet-34 architecture, as described in [9]  The model uses convolution layers with Relu activations to map the spectrogram of size 257x500 input into 3D feature map of size 1x32x512  This feature cube is converted into 2D feature map of dimension 32x512 and fed into Ghost-VLAD/NetVLAD layer to generate a representation that has more language discrimination capacity  We use Adam optimizer with an initial learning rate of 0 01 and a final learning rate of 0 00001 for training  Each model is trained for 15 epochs with early stopping criteria 
For the baseline, we train an i-vector model using GMM-UBM  We fit a small classifier on top of the generated i-vectors to measure the accuracy  This model is referred as i-vector+svm   To compare our model with the previous state of the art system, we set up the x-vector language identification system [8]  The x-vector model used time-delay neural networks (TDNN) along with statistic-pooling  We use 7 layer TDNN architecture similar to [8] for training  We refer to this model as tdnn+stat-pool   Finally, we set up a Deep LSTM based language identification system similar to [4] but with little modification where we add statistics pooling for the last layers hidden activities before classification  We use 3 layer Bi-LSTM with 256 hidden units at each layer  We refer to this model as LSTM+stat-pool  We train our i-vector+svm and TDNN+stat-pool using Kaldi toolkit  We train our NetVLAD and GhostVLAD experiments using Keras by modifying the code given by [9] for language identification  We train the LSTM+stat-pool and the remaining experiments using Pytorch [14] toolkit, and we will opensource all the codes and data soon 
RESULTS
In this section, we compare the performance of our system with the recent state of the art language identification approaches  We also compare different pooling strategies and finally, compare the robustness of our system to the length of the input spectrogram during training  We visualize the embeddings generated by the GhostVLAD method and conclude that the GhostVLAD embeddings shows very good feature discrimination capabilities 
RESULTS ::: Comparison with different approaches
We compare our system performance with the previous state of the art language identification approaches, as shown in Table 2  The i-vector+svm system is trained using GMM-UBM models to generate i-vectors as proposed in [1]  Once the i-vectors are extracted, we fit SVM classifier to classify the audio  The TDNN+stat-pool system is trained with a statistics pooling layer and is called the x-vector system as proposed by David Snyder et al  [11] and is currently the state of the art language identification approach as far as our knowledge  Our methods outperform the state of the art x-vector system by absolute 1 88% improvement in F1-score, as shown in Table 2 
RESULTS ::: Comparison with different pooling techniques
We compare our approach with different pooling strategies in Table 3  We use ResNet as our base feature extraction network  We keep the base network the same and change only the pooling layers to see which pooling approach performs better for language identification task  Our experiments show that GhostVLAD pooling outperforms all the other pooling methods by achieving 98 43% F1-Score 
RESULTS ::: Duration analysis
To observe the performance of our method with different input durations, we conducted an experiment where we train our model on different input durations  Since our model uses ResNet as the base feature extractor, we need to feed fixed-length spectrogram  We conducted 4 different experiments where we trained the model using 2sec, 3sec, 4sec and 5sec spectrograms containing 200,300,400 and 500 frames respectively  We observed that the model trained with a 5sec spectrogram is the best model, as shown in Table 4 
RESULTS ::: Visualization of embeddings
We visualize the embeddings generated by our approach to see the effectiveness  We extracted 512-dimensional embeddings for our testing data and reduced the dimensionality using t-sne projection  The t-sne plot of the embeddings space is shown in Figure 3  The plot shows that the embeddings learned by our approach has very good discriminative properties
Conclusion
In this work, we use Ghost-VLAD pooling approach that was originally proposed for face recognition to improve language identification performance for Indian languages  We collected and curated 630 hrs audio data from news All India Radio news channel for 7 Indian languages  Our experimental results shows that our approach outperforms the previous state of the art methods by an absolute 1 88% F1-score  We have also conducted experiments with different pooling strategies proposed in the past, and the GhostVLAD pooling approach turns out to be the best approach for aggregating frame-level features into a single utterance level feature  Our experiments also prove that our approach works much better even if the input during training contains smaller durations  Finally, we see that the embeddings generated by our method has very good language discriminative features and helps to improve the performance of language identification 

ground_truth: Coverage, Avg. MCC and avg. +ve F1 score

strategy formulation ability, we introduce a measure called Coverage( INLINEFORM0 ), To evaluate the predictive performance, we use Avg. MCC and avg. +ve F1 score

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. which languages are explored?
Context 4: Introduction
This paper describes our approach and results for Task 2 of the CoNLL–SIGMORPHON 2018 shared task on universal morphological reinflection BIBREF0   The task is to generate an inflected word form given its lemma and the context in which it occurs 
Morphological (re)inflection from context is of particular relevance to the field of computational linguistics: it is compelling to estimate how well a machine-learned system can capture the morphosyntactic properties of a word given its context, and map those properties to the correct surface form for a given lemma 
There are two tracks of Task 2 of CoNLL–SIGMORPHON 2018: in Track 1 the context is given in terms of word forms, lemmas and morphosyntactic descriptions (MSD); in Track 2 only word forms are available  See Table TABREF1 for an example  Task 2 is additionally split in three settings based on data size: high, medium and low, with high-resource datasets consisting of up to 70K instances per language, and low-resource datasets consisting of only about 1K instances 
The baseline provided by the shared task organisers is a seq2seq model with attention (similar to the winning system for reinflection in CoNLL–SIGMORPHON 2016, BIBREF1 ), which receives information about context through an embedding of the two words immediately adjacent to the target form  We use this baseline implementation as a starting point and achieve the best overall accuracy of 49 87 on Task 2 by introducing three augmentations to the provided baseline system: (1) We use an LSTM to encode the entire available context; (2) We employ a multi-task learning approach with the auxiliary objective of MSD prediction; and (3) We train the auxiliary component in a multilingual fashion, over sets of two to three languages 
In analysing the performance of our system, we found that encoding the full context improves performance considerably for all languages: 11 15 percentage points on average, although it also highly increases the variance in results  Multi-task learning, paired with multilingual training and subsequent monolingual finetuning, scored highest for five out of seven languages, improving accuracy by another 9 86% on average 
System Description
Our system is a modification of the provided CoNLL–SIGMORPHON 2018 baseline system, so we begin this section with a reiteration of the baseline system architecture, followed by a description of the three augmentations we introduce 
Baseline
The CoNLL–SIGMORPHON 2018 baseline is described as follows:
The system is an encoder-decoder on character sequences  It takes a lemma as input and generates a word form  The process is conditioned on the context of the lemma [   ] The baseline treats the lemma, word form and MSD of the previous and following word as context in track 1  In track 2, the baseline only considers the word forms of the previous and next word  [   ] The baseline system concatenates embeddings for context word forms, lemmas and MSDs into a context vector  The baseline then computes character embeddings for each character in the input lemma  Each of these is concatenated with a copy of the context vector  The resulting sequence of vectors is encoded using an LSTM encoder  Subsequently, an LSTM decoder generates the characters in the output word form using encoder states and an attention mechanism 
To that we add a few details regarding model size and training schedule:
the number of LSTM layers is one;
embedding size, LSTM layer size and attention layer size is 100;
models are trained for 20 epochs;
on every epoch, training data is subsampled at a rate of 0 3;
LSTM dropout is applied at a rate 0 3;
context word forms are randomly dropped at a rate of 0 1;
the Adam optimiser is used, with a default learning rate of 0 001; and
trained models are evaluated on the development data (the data for the shared task comes already split in train and dev sets) 
Our system
Here we compare and contrast our system to the baseline system  A diagram of our system is shown in Figure FIGREF4  
The idea behind this modification is to provide the encoder with access to all morpho-syntactic cues present in the sentence  In contrast to the baseline, which only encodes the immediately adjacent context of a target word, we encode the entire context  All context word forms, lemmas, and MSD tags (in Track 1) are embedded in their respective high-dimensional spaces as before, and their embeddings are concatenated  However, we now reduce the entire past context to a fixed-size vector by encoding it with a forward LSTM, and we similarly represent the future context by encoding it with a backwards LSTM 
We introduce an auxiliary objective that is meant to increase the morpho-syntactic awareness of the encoder and to regularise the learning process—the task is to predict the MSD tag of the target form  MSD tag predictions are conditioned on the context encoding, as described in UID15   Tags are generated with an LSTM one component at a time, e g  the tag PRO;NOM;SG;1 is predicted as a sequence of four components, INLINEFORM0 PRO, NOM, SG, 1 INLINEFORM1  
For every training instance, we backpropagate the sum of the main loss and the auxiliary loss without any weighting 
As MSD tags are only available in Track 1, this augmentation only applies to this track 
The parameters of the entire MSD (auxiliary-task) decoder are shared across languages 
Since a grouping of the languages based on language family would have left several languages in single-member groups (e g  Russian is the sole representative of the Slavic family), we experiment with random groupings of two to three languages  Multilingual training is performed by randomly alternating between languages for every new minibatch  We do not pass any information to the auxiliary decoder as to the source language of the signal it is receiving, as we assume abstract morpho-syntactic features are shared across languages 
After 20 epochs of multilingual training, we perform 5 epochs of monolingual finetuning for each language  For this phase, we reduce the learning rate to a tenth of the original learning rate, i e  0 0001, to ensure that the models are indeed being finetuned rather than retrained 
We keep all hyperparameters the same as in the baseline

ground_truth: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish

Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish , Swedish

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which 7 Indian languages do they experiment with?
Context 4:  Training data is split 90:10 for training and validation  We train our models for 50 epochs, adding early stopping with a tolerance of five epochs of no improvement in the validation loss  We do not subsample from the training data 
We train models for 50 different random combinations of two to three languages in Track 1, and 50 monolingual models for each language in Track 2  Instead of picking the single model that performs best on the development set and thus risking to select a model that highly overfits that data, we use an ensemble of the five best models, and make the final prediction for a given target form with a majority vote over the five predictions 
Results and Discussion
Test results are listed in Table TABREF17   Our system outperforms the baseline for all settings and languages in Track 1 and for almost all in Track 2—only in the high resource setting is our system not definitively superior to the baseline 
Interestingly, our results in the low resource setting are often higher for Track 2 than for Track 1, even though contextual information is less explicit in the Track 2 data and the multilingual multi-tasking approach does not apply to this track  We interpret this finding as an indicator that a simpler model with fewer parameters works better in a setting of limited training data  Nevertheless, we focus on the low resource setting in the analysis below due to time limitations  As our Track 1 results are still substantially higher than the baseline results, we consider this analysis valid and insightful 
Ablation Study
We analyse the incremental effect of the different features in our system, focusing on the low-resource setting in Track 1 and using development data 
Encoding the entire context with an LSTM highly increases the variance of the observed results  So we trained fifty models for each language and each architecture  Figure FIGREF23 visualises the means and standard deviations over the trained models  In addition, we visualise the average accuracy for the five best models for each language and architecture, as these are the models we use in the final ensemble prediction  Below we refer to these numbers only 
The results indicate that encoding the full context with an LSTM highly enhances the performance of the model, by 11 15% on average  This observation explains the high results we obtain also for Track 2 
Adding the auxiliary objective of MSD prediction has a variable effect: for four languages (de, en, es, and sv) the effect is positive, while for the rest it is negative  We consider this to be an issue of insufficient data for the training of the auxiliary component in the low resource setting we are working with 
We indeed see results improving drastically with the introduction of multilingual training, with multilingual results being 7 96% higher than monolingual ones on average 
We studied the five best models for each language as emerging from the multilingual training (listed in Table TABREF27 ) and found no strong linguistic patterns  The en–sv pairing seems to yield good models for these languages, which could be explained in terms of their common language family and similar morphology  The other natural pairings, however, fr–es, and de–sv, are not so frequent among the best models for these pairs of languages 
Finally, monolingual finetuning improves accuracy across the board, as one would expect, by 2 72% on average 
The final observation to be made based on this breakdown of results is that the multi-tasking approach paired with multilingual training and subsequent monolingual finetuning outperforms the other architectures for five out of seven languages: de, en, fr, ru and sv  For the other two languages in the dataset, es and fi, the difference between this approach and the approach that emerged as best for them is less than 1%  The overall improvement of the multilingual multi-tasking approach over the baseline is 18 30% 
Error analysis
Here we study the errors produced by our system on the English test set to better understand the remaining shortcomings of the approach  A small portion of the wrong predictions point to an incorrect interpretation of the morpho-syntactic conditioning of the context, e g  the system predicted plan instead of plans in the context Our _ include raising private capital  The majority of wrong predictions, however, are nonsensical, like bomb for job, fify for fixing, and gnderrate for understand  This observation suggests that generally the system did not learn to copy the characters of lemma into inflected form, which is all it needs to do in a large number of cases  This issue could be alleviated with simple data augmentation techniques that encourage autoencoding BIBREF2  
MSD prediction
Figure FIGREF32 summarises the average MSD-prediction accuracy for the multi-tasking experiments discussed above  Accuracy here is generally higher than on the main task, with the multilingual finetuned setup for Spanish and the monolingual setup for French scoring best: 66 59% and 65 35%, respectively  This observation illustrates the added difficulty of generating the correct surface form even when the morphosyntactic description has been identified correctly 
We observe some correlation between these numbers and accuracy on the main task: for de, en, ru and sv, the brown, pink and blue bars here pattern in the same way as the corresponding INLINEFORM0 's in Figure FIGREF23   One notable exception to this pattern is fr where inflection gains a lot from multilingual training, while MSD prediction suffers greatly  Notice that the magnitude of change is not always the same, however, even when the general direction matches: for ru, for example, multilingual training benefits inflection much more than in benefits MSD prediction, even though the MSD decoder is the only component that is actually shared between languages  This observation illustrates the two-fold effect of multi-task training: an auxiliary task can either inform the main task through the parameters the two tasks share, or it can help the main task learning through its regularising effect 
Related Work
Our system is inspired by previous work on multi-task learning and multi-lingual learning, mainly building on two intuitions: (1) jointly learning related tasks tends to be beneficial BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 ; and (2) jointly learning related languages in an MTL-inspired framework tends to be beneficial BIBREF8 , BIBREF9 , BIBREF10   In the context of computational morphology, multi-lingual approaches have previously been employed for morphological reinflection BIBREF2 and for paradigm completion BIBREF11   In both of these cases, however, the available datasets covered more languages, 40 and 21, respectively, which allowed for linguistically-motivated language groupings and for parameter sharing directly on the level of characters  BIBREF10 explore parameter sharing between related languages for dependency parsing, and find that sharing is more beneficial in the case of closely related languages 
Conclusions
In this paper we described our system for the CoNLL–SIGMORPHON 2018 shared task on Universal Morphological Reinflection, Task 2, which achieved the best performance out of all systems submitted, an overall accuracy of 49 87  We showed in an ablation study that this is due to three core innovations, which extend a character-based encoder-decoder model: (1) a wide context window, encoding the entire available context; (2) multi-task learning with the auxiliary task of MSD prediction, which acts as a regulariser; (3) a multilingual approach, exploiting information across languages  In future work we aim to gain better understanding of the increase in variance of the results introduced by each of our modifications and the reasons for the varying effect of multi-task learning for different languages 
Acknowledgements
We gratefully acknowledge the support of the NVIDIA Corporation with the donation of the Titan Xp GPU used for this research 

ground_truth: Hindi, English, Kannada, Telugu, Assamese, Bengali and Malayalam

Kannada, Hindi, Telugu, Malayalam, Bengali,  English and Assamese (in table, missing in text)