prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what dataset is used in this paper?
Context 1: Introduction
Users of photo-sharing websites such as Flickr often provide short textual descriptions in the form of tags to help others find the images  With the availability of GPS systems in current electronic devices such as smartphones, latitude and longitude coordinates are nowadays commonly made available as well  The tags associated with such georeferenced photos often describe the location where these photos were taken, and Flickr can thus be regarded as a source of environmental information  The use of Flickr for modelling urban environments has already received considerable attention  For instance, various approaches have been proposed for modelling urban regions BIBREF0 , and for identifying points-of-interest BIBREF1 and itineraries BIBREF2 , BIBREF3   However, the usefulness of Flickr for characterizing the natural environment, which is the focus of this paper, is less well-understood 
Many recent studies have highlighted that Flickr tags capture valuable ecological information, which can be used as a complementary source to more traditional sources  To date, however, ecologists have mostly used social media to conduct manual evaluations of image content with little automated exploitation of the associated tags BIBREF4 , BIBREF5 , BIBREF6   One recent exception is BIBREF7 , where bag-of-words representations derived from Flickr tags were found to give promising result for predicting a range of different environemental phenomena 
Our main hypothesis in this paper is that by using vector space embeddings instead of bag-of-words representations, the ecological information which is implicitly captured by Flickr tags can be utilized in a more effective way  Vector space embeddings are representations in which the objects from a given domain are encoded using relatively low-dimensional vectors  They have proven useful in natural language processing, especially for encoding word meaning BIBREF8 , BIBREF9 , and in machine learning more generally  In this paper, we are interested in the use of such representations for modelling geographic locations  Our main motivation for using vector space embeddings is that they allow us to integrate the textual information we get from Flickr with available structured information in a very natural way  To this end, we rely on an adaptation of the GloVe word embedding model BIBREF9 , but rather than learning word vectors, we learn vectors representing locations  Similar to how the representation of a word in GloVe is determined by the context words surrounding it, the representation of a location in our model is determined by the tags of the photos that have been taken near that location  To incorporate numerical features from structured environmental datasets (e g  average temperature), we associate with each such feature a linear mapping that can be used to predict that feature from a given location vector  This is inspired by the fact that salient properties of a given domain can often be modelled as directions in vector space embeddings BIBREF10 , BIBREF11 , BIBREF12   Finally, evidence from categorical datasets (e g  land cover types) is taken into account by requiring that locations belonging to the same category are represented using similar vectors, similar to how semantic types are sometimes modelled in the context of knowledge graph embedding BIBREF13  
While our point-of-departure is a standard word embedding model, we found that the off-the-shelf GloVe model performed surprisingly poorly, meaning that a number of modifications are needed to achieve good results  Our main findings are as follows  First, given that the number of tags associated with a given location can be quite small, it is important to apply some kind of spatial smoothing, i e  the importance of a given tag for a given location should not only depend on the occurrences of the tag at that location, but also on its occurrences at nearby locations  To this end, we use a formulation which is based on spatially smoothed version of pointwise mutual information  Second, given the wide diversity in the kind of information that is covered by Flickr tags, we find that term selection is in some cases critical to obtain vector spaces that capture the relevant aspects of geographic locations  For instance, many tags on Flickr refer to photography related terms, which we would normally not want to affect the vector representation of a given location  Finally, even with these modifications, vector space embeddings learned from Flickr tags alone are sometimes outperformed by bag-of-words representations  However, our vector space embeddings lead to substantially better predictions in cases where structured (scientific) information is also taken into account  In this sense, the main value of using vector space embeddings in this context is not so much about abstracting away from specific tag usages, but rather about the fact that such representations allow us to integrate numerical and categorical features in a much more natural way than is possible with bag-of-words representations 
The remainder of this paper is organized as follows  In the next section, we provide a discussion of existing work  Section SECREF3 then presents our model for embedding geographic locations from Flickr tags and structured data  Next, in Section SECREF4 we provide a detailed discussion about the experimental results  Finally, Section SECREF5 summarizes our conclusions 
Vector space embeddings
The use of low-dimensional vector space embeddings for representing objects has already proven effective in a large number of applications, including natural language processing (NLP), image processing, and pattern recognition  In the context of NLP, the most prominent example is that of word embeddings, which represent word meaning using vectors of typically around 300 dimensions  A large number of different methods for learning such word embeddings have already been proposed, including Skip-gram and the Continuous Bag-of-Words (CBOW) model BIBREF8 , GloVe BIBREF9 , and fastText BIBREF14   They have been applied effectively in many downstream NLP tasks such as sentiment analysis BIBREF15 , part of speech tagging BIBREF16 , BIBREF17 , and text classification BIBREF18 , BIBREF19   The model we consider in this paper builds on GloVe, which was designed to capture linear regularities of word-word co-occurrence  In GloVe, there are two word vectors INLINEFORM0 and INLINEFORM1 for each word in the vocabulary, which are learned by minimizing the following objective: DISPLAYFORM0
where INLINEFORM0 is the number of times that word INLINEFORM1 appears in the context of word INLINEFORM2 , INLINEFORM3 is the vocabulary size, INLINEFORM4 is the target word bias, INLINEFORM5 is the context word bias  The weighting function INLINEFORM6 is used to limit the impact of rare terms  It is defined as 1 if INLINEFORM7 and as INLINEFORM8 otherwise, where INLINEFORM9 is usually fixed to 100 and INLINEFORM10 to 0 75  Intuitively, the target word vectors INLINEFORM11 correspond to the actual word representations which we would like to find, while the context word vectors INLINEFORM12 model how occurrences of INLINEFORM13 in the context of a given word INLINEFORM14 affect the representation of this latter word  In this paper we will use a similar model, which will however be aimed at learning location vectors instead of the target word vectors 
Beyond word embeddings, various methods have been proposed for learning vector space representations from structured data such as knowledge graphs BIBREF20 , BIBREF21 , BIBREF22 , social networks BIBREF23 , BIBREF24 and taxonomies BIBREF25 , BIBREF26   The idea of combining a word embedding model with structured information has also been explored by several authors, for example to improve the word embeddings based on information coming from knowledge graphs BIBREF27 , BIBREF28   Along similar lines, various lexicons have been used to obtain word embeddings that are better suited at modelling sentiment BIBREF15 and antonymy BIBREF29 , among others  The method proposed by BIBREF30 imposes the condition that words that belong to the same semantic category are closer together than words from different categories, which is somewhat similar in spirit to how we will model categorical datasets in our model 
Embeddings for geographic information
The problem of representing geographic locations using embeddings has also attracted some attention  An early example is BIBREF31 , which used principal component analysis and stacked autoencoders to learn low-dimensional vector representations of city neighbourhoods based on census data  They use these representations to predict attributes such as crime, which is not included in the given census data, and find that in most of the considered evaluation tasks, the low-dimensional vector representations lead to more faithful predictions than the original high-dimensional census data 
Some existing works combine word embedding models with geographic coordinates  For example, in BIBREF32 an approach is proposed to learn word embeddings based on the assumption that words which tend to be used in the same geographic locations are likely to be similar  Note that their aim is dual to our aim in this paper: while they use geographic location to learn word vectors, we use textual descriptions to learn vectors representing geographic locations 
Several methods also use word embedding models to learn representations of Points-of-Interest (POIs) that can be used for predicting user visits BIBREF33 , BIBREF34 , BIBREF35   These works use the machinery of existing word embedding models to learn POI representations, intuitively by letting sequences of POI visits by a user play the role of sequences of words in a sentence  In other words, despite the use of word embedding models, many of these approaches do not actually consider any textual information  For example, in BIBREF34 the Skip-gram model is utilized to create a global pattern of users' POIs  Each location was treated as a word and the other locations visited before or after were treated as context words  They then use a pair-wise ranking loss BIBREF36 which takes into account the user's location visit frequency to personalize the location recommendations  The methods of BIBREF34 were extended in BIBREF35 to use a temporal embedding and to take more account of geographic context, in particular the distances between preferred and non-preferred neighboring POIs, to create a “geographically hierarchical pairwise preference ranking model”  Similarly, in BIBREF37 the CBOW model was trained with POI data  They ordered POIs spatially within the traffic-based zones of urban areas  The ordering was used to generate characteristic vectors of POI types  Zone vectors represented by averaging the vectors of the POIs contained in them, were then used as features to predict land use types  In the CrossMap method BIBREF38 they learned embeddings for spatio-temporal hotspots obtained from social media data of locations, times and text  In one form of embedding, intended to enable reconstruction of records, neighbourhood relations in space and time were encoded by averaging hotspots in a target location's spatial and temporal neighborhoods  They also proposed a graph-based embedding method with nodes of location, time and text  The concatenation of the location, time and text vectors were then used as features to predict peoples' activities in urban environments  Finally, in BIBREF39 , a method is proposed that uses the Skip-gram model to represent POI types, based on the intuition that the vector representing a given POI type should be predictive of the POI types that found near places of that type 
Our work is different from these studies, as our focus is on representing locations based on a given text description of that location (in the form of Flickr tags), along with numerical and categorical features from scientific datasets 
Analyzing Flickr tags
Many studies have focused on analyzing Flickr tags to extract useful information in domains such as linguistics BIBREF40 , geography BIBREF0 , BIBREF41 , and ecology BIBREF42 , BIBREF7 , BIBREF43   Most closely related to our work, BIBREF7 found that the tags of georeferenced Flickr photos can effectively supplement traditional scientific environmental data in tasks such as predicting climate features, land cover, species occurrence, and human assessments of scenicness  To encode locations, they simply combine a bag-of-words representation of geographically nearby tags with a feature vector that encodes associated structured scientific data  They found that the predictive value of Flickr tags is roughly on a par with that of the scientific datasets, and that combining both types of information leads to significantly better results than using either of them alone  As we show in this paper, however, their straightforward way of combining both information sources, by concatenating the two types of feature vectors, is far from optimal 
Despite the proven importance of Flickr tags, the problem of embedding Flickr tags has so far received very limited attention  To the best of our knowledge, BIBREF44 is the only work that generated embeddings for Flickr tags  However, their focus was on learning embeddings that capture word meaning (being evaluated on word similarity tasks), whereas we use such embeddings as part of our method for representing locations 
Model Description
In this section, we introduce our embedding model, which combines Flickr tags and structured scientific information to represent a set of locations INLINEFORM0   The proposed model has the following form: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are parameters to control the importance of each component in the model  Component INLINEFORM2 will be used to constrain the representation of the locations based on their textual description (i e  Flickr tags), INLINEFORM3 will be used to constrain the representation of the locations based on their numerical features, and INLINEFORM4 will impose the constraint that locations belonging to the same category should be close together in the space  We will discuss each of these components in more detail in the following sections 
Tag Based Location Embedding
Many of the tags associated with Flickr photos describe characteristics of the places where these photos were taken BIBREF45 , BIBREF46 , BIBREF47   For example, tags may correspond to place names (e g  Brussels, England, Scandinavia), landmarks (e g  Eiffel Tower, Empire State Building) or land cover types (e g  mountain, forest, beach)  To allow us to build location models using such tags, we collected the tags and meta-data of 70 million Flickr photos with coordinates in Europe (which is the region our experiments will focus on), all of which were uploaded to Flickr before the end of September 2015  In this section we first explain how tags can be weighted to obtain bag-of-words representations of locations from Flickr  Subsequently we describe a tag selection method, which will allow us to specialize the embedding depending on which aspects of the considered locations are of interest, after which we discuss the actual embedding model 
Tag weighting  Let INLINEFORM0 be a set of geographic locations, each characterized by latitude and longitude coordinates  To generate a bag-of-words representation of a given location, we have to weight the relevance of each tag to that location  To this end, we have followed the weighting scheme from BIBREF7 , which combines a Gaussian kernel (to model spatial proximity) with Positive Pointwise Mutual Information (PPMI) BIBREF48 , BIBREF49  
Let us write INLINEFORM0 for the set of users who have assigned tag INLINEFORM1 to a photo with coordinates near INLINEFORM2   To assess how relevant INLINEFORM3 is to the location INLINEFORM4 , the number of times INLINEFORM5 occurs in photos near INLINEFORM6 is clearly an important criterion  However, rather than simply counting the number of occurrences within some fixed radius, we use a Gaussian kernel to weight the tag occurrences according to their distance from that location: INLINEFORM7
where the threshold INLINEFORM0 is assumed to be fixed, INLINEFORM1 is the location of a Flickr photo, INLINEFORM2 is the Haversine distance, and we will assume that the bandwidth parameter INLINEFORM3 is set to INLINEFORM4   A tag occurrence is counted only once for all photos by the same user at the same location, which is important to reduce the impact of bulk uploading  The value INLINEFORM5 reflects how frequent tag INLINEFORM6 is near location INLINEFORM7 , but it does not yet take into account the total number of tag occurrences near INLINEFORM8 , nor how popular the tag INLINEFORM9 is overall  To measure how strongly tag INLINEFORM10 is associated with location INLINEFORM11 , we use PPMI, which is a commonly used measure of association in natural language processing  However, rather than estimating PPMI scores from term frequencies, we will use the INLINEFORM12 values instead: INLINEFORM13
where: INLINEFORM0
with INLINEFORM0 the set of all tags, and INLINEFORM1 the set of locations 
Tag selection  Inspired by BIBREF50 , we use a term selection method in order to focus on the tags that are most important for the tasks that we want to consider and reduce the impact of tags that might relate only to a given individual or a group of users  In particular, we obtained good results with a method based on Kullback-Leibler (KL) divergence, which is based on BIBREF51   Let INLINEFORM0 be a set of (mutually exclusive) properties of locations in which we are interested (e g  land cover categories)  For the ease of presentation, we will identify INLINEFORM1 with the set of locations that have the corresponding property  Then, we select tags from INLINEFORM2 that maximize the following score: INLINEFORM3
where INLINEFORM0 is the probability that a photo with tag INLINEFORM1 has a location near INLINEFORM2 and INLINEFORM3 is the probability that an arbitrary tag occurrence is assigned to a photo near a location in INLINEFORM4   Since INLINEFORM5 often has to be estimated from a small number of tag occurrences, it is estimated using Bayesian smoothing: INLINEFORM6
where INLINEFORM0 is a parameter controlling the amount of smoothing, which will be tuned in the experiments  On the other hand, for INLINEFORM1 we can simply use a maximum likelihood estimation: INLINEFORM2
Location embedding  We now want to find a vector INLINEFORM0 for each location INLINEFORM1 such that similar locations are represented using similar vectors  To achieve this, we use a close variant of the GloVe model, where tag occurrences are treated as context words of geographic locations  In particular, with each location INLINEFORM2 we associate a vector INLINEFORM3 and with each tag INLINEFORM4 we associate a vector INLINEFORM5 and a bias term INLINEFORM6 , and consider the following objective (which in our full model ( EQREF7 ) will be combined with components that are derived from the structured information): INLINEFORM7
Note how tags play the role of the context words in the GloVe model, while instead of learning target word vectors we now learn location vectors  In contrast to GloVe, our objective does not directly refer to co-occurrence statistics, but instead uses the INLINEFORM0 scores  One important consequence of this is that we can also consider pairs INLINEFORM1 for which INLINEFORM2 does not occur in INLINEFORM3 at all; such pairs are usually called negative examples  While they cannot be used in the standard GloVe model, some authors have already reported that introducing negative examples in variants of GloVe can lead to an improvement BIBREF52   In practice, evaluating the full objective above would not be computationally feasible, as we may need to consider millions of locations and millions of tags  Therefore, rather than considering all tags in INLINEFORM4 for the inner summation, we only consider those tags that appear at least once near location INLINEFORM5 together with a sample of negative examples 
Structured Environmental Data
There is a wide variety of structured data that can be used to describe locations  In this work, we have restricted ourselves to the same datasets as BIBREF7   These include nine (real-valued) numerical features, which are latitude, longitude, elevation, population, and five climate related features (avg  temperature, avg  precipitation, avg  solar radiation, avg  wind speed, and avg  water vapor pressure)  In addition, 180 categorical features were used, which are CORINE land cover classes at level 1 (5 classes), level 2 (15 classes) and level 3 (44 classes) and 116 soil types (SoilGrids)  Note that each location should belong to exactly 4 categories: one CORINE class at each of the three levels and a soil type 
Numerical features  Numerical features can be treated similarly to the tag occurrences, i e  we will assume that the value of a given numerical feature can be predicted from the location vectors using a linear mapping  In particular, for each numerical feature INLINEFORM0 we consider a vector INLINEFORM1 and a bias term INLINEFORM2 , and the following objective: INLINEFORM3
where we write INLINEFORM0 for set of all numerical features and INLINEFORM1 is the value of feature INLINEFORM2 for location INLINEFORM3 , after z-score normalization 
Categorical features  To take into account the categorical features, we impose the constraint that locations belonging to the same category should be close together in the space  To formalize this, we represent each category type INLINEFORM0 as a vector INLINEFORM1 , and consider the following objective: INLINEFORM2
Evaluation Tasks
We will use the method from BIBREF7 as our main baseline  This will allow us to directly evaluate the effectiveness of embeddings for the considered problem, since we have used the same structured datasets and same tag weighting scheme  For this reason, we will also follow their evaluation methodology  In particular, we will consider three evaluation tasks:
Predicting the distribution of 100 species across Europe, using the European network of nature protected sites Natura 2000 dataset as ground truth  For each of these species, a binary classification problem is considered  The set of locations INLINEFORM0 is defined as the 26,425 distinct sites occurring in the dataset 
Predicting soil type, again each time treating the task as a binary classification problem, using the same set of locations INLINEFORM0 as in the species distribution experiments  For these experiments, none of the soil type features are used for generating the embeddings 
Predicting CORINE land cover classes at levels 1, 2 and level 3, each time treating the task as a binary classification problem, using the same set of locations INLINEFORM0 as in the species distribution experiments  For these experiments, none of the CORINE features are used for generating the embeddings 
In addition, we will also consider the following regression tasks:
Predicting 5 climate related features: the average precipitation, temperature, solar radiation, water vapor pressure, and wind speed  We again use the same set of locations INLINEFORM0 as for species distribution in this experiment  None of the climate features is used for constructing the embeddings for this experiment 
Predicting people's subjective opinions of landscape beauty in Britain, using the crowdsourced dataset from the ScenicOrNot website as ground truth  The set INLINEFORM0 is chosen as the set of locations of 191 605 rated locations from the ScenicOrNot dataset for which at least one georeferenced Flickr photo exists within a 1 km radius 
Experimental Setup
In all experiments, we use Support Vector Machines (SVMs) for classification problems and Support Vector Regression (SVR) for regression problems to make predictions from our representations of geographic locations  In both cases, we used the SVM INLINEFORM0 implementation BIBREF53   For each experiment, the set of locations INLINEFORM1 was split into two-thirds for training, one-sixth for testing, and one-sixth for tuning the parameters  All embedding models are learned with Adagrad using 30 iterations  The number of dimensions is chosen for each experiment from INLINEFORM2 based on the tuning data  For the parameters of our model in Equation EQREF7 , we considered values of INLINEFORM3 from {0 1, 0 01, 0 001, 0 0001} and values of INLINEFORM4 from {1, 10, 100, 1000, 10 000, 100 000}  To compute KL divergence, we need to determine a set of classes INLINEFORM5 for each experiment  For classification problems, we can simply consider the given categories, but for the regression problems we need to define such classes by discretizing the numerical values  For the scenicness experiments, we considered scores 3 and 7 as cut-off points, leading to three classes (i e  less than 3, between 3 and 7, and above 7)  Similarly, for each climate related features, we consider two cut-off values for discretization: 5 and 15 for average temperature, 50 and 100 for average precipitation, 10 000 and 17 000 for average solar radiation, 0 7 and 1 for average water vapor pressure, and 3 and 5 for wind speed  The smoothing parameter INLINEFORM6 was selected among INLINEFORM7 based on the tuning data  In all experiments where term selection is used, we select the top 100 000 tags  We fixed the radius INLINEFORM8 at 1km when counting the number of tag occurrences  Finally, we set the number of negative examples as 10 times the number of positive examples for each location, but with a cap at 1000 negative examples in each region for computational reasons  We tune all parameters with respect to the F1 score for the classification tasks, and Spearman INLINEFORM9 for the regression tasks 
Variants and Baseline Methods
We will refer to our model as EGEL (Embedding GEographic Locations), and will consider the following variants  EGEL-Tags only uses the information from the Flickr tags (i e  component INLINEFORM0 ), without using any negative examples and without feature selection  EGEL-Tags+NS is similar to EGEL-Tags but with the addition of negative examples  EGEL-KL(Tags+NS) additionally considers term selection  EGEL-All is our full method, i e  it additionally uses the structured information  We also consider the following baselines  BOW-Tags represents locations using a bag-of-words representation, using the same tag weighting as the embedding model  BOW-KL(Tags) uses the same representation but after term selection, using the same KL-based method as the embedding model  BOW-All combines the bag-of-words representation with the structured information, encoded as proposed in BIBREF7   GloVe uses the objective from the original GloVe model for learning location vectors, i e  this variant differs from EGEL-Tags in that instead of INLINEFORM1 we use the number of co-occurrences of tag INLINEFORM2 near location INLINEFORM3 , measured as INLINEFORM4  
Results and Discussion
We present our results for the binary classification tasks in Tables TABREF23 – TABREF24 in terms of average precision, average recall and macro average F1 score  The results of the regression tasks are reported in Tables TABREF25 and TABREF29 in terms of the mean absolute error between the predicted and actual scores, as well as the Spearman INLINEFORM0 correlation between the rankings induced by both sets of scores  It can be clearly seen from the results that our proposed method (EGEL-All) can effectively integrate Flickr tags with the available structured information  It outperforms the baselines for all the considered tasks  Furthermore, note that the PPMI-based weighting in EGEL-Tags consistently outperforms GloVe and that both the addition of negative examples and term selection lead to further improvements  The use of term selection leads to particularly substantial improvements for the regression problems 
While our experimental results confirm the usefulness of embeddings for predicting environmental features, this is only consistently the case for the variants that use both the tags and the structured datasets  In particular, comparing BOW-Tags with EGEL-Tags, we sometimes see that the former achieves the best results  While this might seem surprising, it is in accordance with the findings in BIBREF54 , BIBREF38 , among others, where it was also found that bag-of-words representations can sometimes lead to surprisingly effective baselines  Interestingly, we note that in all cases where EGEL-KL(Tags+NS) performs worse than BOW-Tags, we also find that BOW-KL(Tags) performs worse than BOW-Tags  This suggests that for these tasks there is a very large variation in the kind of tags that can inform the prediction model, possibly including e g  user-specific tags  Some of the information captured by such highly specific but rare tags is likely to be lost in the embedding 
To further analyze the difference in performance between BoW representations and embeddings, Figure TABREF29 compares the performance of the GloVe model with the bag-of-words model for predicting place scenicness, as a function of the number of tag occurrences at the considered locations  What is clearly noticeable in Figure TABREF29 is that GloVe performs better than the bag-of-words model for large corpora and worse for smaller corpora  This issue has been alleviated in our embedding method by the addition of negative examples 
 Conclusions
In this paper, we have proposed a model to learn geographic location embeddings using Flickr tags, numerical environmental features, and categorical information  The experimental results show that our model can integrate Flickr tags with structured information in a more effective way than existing methods, leading to substantial improvements over baseline methods on various prediction tasks about the natural environment 
Acknowledgments
Shelan Jeawak has been sponsored by HCED Iraq  Steven Schockaert has been supported by ERC Starting Grant 637277 
Context 2: Introduction
Chatbots such as dialog and question-answering systems have a long history in AI and natural language processing  Early such systems were mostly built using markup languages such as AIML, handcrafted conversation generation rules, and/or information retrieval techniques BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   Recent neural conversation models BIBREF4 , BIBREF5 , BIBREF6 are even able to perform open-ended conversations  However, since they do not use explicit knowledge bases and do not perform inference, they often suffer from generic and dull responses BIBREF5 , BIBREF7   More recently, BIBREF8 and BIBREF9 proposed to use knowledge bases (KBs) to help generate responses for knowledge-grounded conversation  However, one major weakness of all existing chat systems is that they do not explicitly or implicitly learn new knowledge in the conversation process  This seriously limits the scope of their applications  In contrast, we humans constantly learn new knowledge in our conversations  Even if some existing systems can use very large knowledge bases either harvested from a large data source such as the Web or built manually, these KBs still miss a large number of facts (knowledge) BIBREF10   It is thus important for a chatbot to continuously learn new knowledge in the conversation process to expand its KB and to improve its conversation ability 
In recent years, researchers have studied the problem of KB completion, i e , inferring new facts (knowledge) automatically from existing facts in a KB  KB completion (KBC) is defined as a binary classification problem: Given a query triple, ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ), we want to predict whether the source entity INLINEFORM3 and target entity INLINEFORM4 can be linked by the relation INLINEFORM5   However, existing approaches BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 solve this problem under the closed-world assumption, i e , INLINEFORM6 , INLINEFORM7 and INLINEFORM8 are all known to exist in the KB  This is a major weakness because it means that no new knowledge or facts may contain unknown entities or relations  Due to this limitation, KBC is clearly not sufficient for knowledge learning in conversations because in a conversation, the user can say anything, which may contain entities and relations that are not already in the KB 
In this paper, we remove this assumption of KBC, and allow all INLINEFORM0 , INLINEFORM1 and INLINEFORM2 to be unknown  We call the new problem open-world knowledge base completion (OKBC)  OKBC generalizes KBC  Below, we show that solving OKBC naturally provides the ground for knowledge learning and inference in conversations  In essence, we formulate an abstract problem of knowledge learning and inference in conversations as a well-defined OKBC problem in the interactive setting 
From the perspective of knowledge learning in conversations, essentially we can extract two key types of information, true facts and queries, from the user utterances  Queries are facts whose truth values need to be determined  Note that we do not study fact or relation extraction in this paper as there is an extensive work on the topic  (1) For a true fact, we will incorporate it into the KB  Here we need to make sure that it is not already in the KB, which involves relation resolution and entity linking  After a fact is added to the KB, we may predict that some related facts involving some existing relations in the KB may also be true (not logical implications as they can be automatically inferred)  For example, if the user says “Obama was born in USA,” the system may guess that (Obama, CitizenOf, USA) (meaning that Obama is a citizen of USA) could also be true based on the current KB  To verify this fact, it needs to solve a KBC problem by treating (Obama, CitizenOf, USA) as a query  This is a KBC problem because the fact (Obama, BornIn, USA) extracted from the original sentence has been added to the KB  Then Obama and USA are in the KB  If the KBC problem is solved, it learns a new fact (Obama, CitizenOf, USA) in addition to the extracted fact (Obama, BornIn, USA)  (2) For a query fact, e g , (Obama, BornIn, USA) extracted from the user question “Was Obama born in USA?” we need to solve the OKBC problem if any of “Obama, “BornIn”, or “USA" is not already in the KB 
We can see that OKBC is the core of a knowledge learning engine for conversation  Thus, in this paper, we focus on solving it  We assume that other tasks such as fact/relation extraction and resolution and guessing of related facts of an extracted fact are solved by other sub-systems 
We solve the OKBC problem by mimicking how humans acquire knowledge and perform reasoning in an interactive conversation  Whenever we encounter an unknown concept or relation while answering a query, we perform inference using our existing knowledge  If our knowledge does not allow us to draw a conclusion, we typically ask questions to others to acquire related knowledge and use it in inference  The process typically involves an inference strategy (a sequence of actions), which interleaves a sequence of processing and interactive actions  A processing action can be the selection of related facts, deriving inference chain, etc , that advances the inference process  An interactive action can be deciding what to ask, formulating a suitable question, etc , that enable us to interact  The process helps grow the knowledge over time and the gained knowledge enables us to communicate better in the future  We call this lifelong interactive learning and inference (LiLi)  Lifelong learning is reflected by the facts that the newly acquired facts are retained in the KB and used in inference for future queries, and that the accumulated knowledge in addition to the updated KB including past inference performances are leveraged to guide future interaction and learning  LiLi should have the following capabilities:
This setting is ideal for many NLP applications like dialog and question-answering systems that naturally provide the scope for human interaction and demand real-time inference 
LiLi starts with the closed-world KBC approach path-ranking (PR) BIBREF11 , BIBREF17 and extends KBC in a major way to open-world knowledge base completion (OKBC)  For a relation INLINEFORM0 , PR works by enumerating paths (except single-link path INLINEFORM1 ) between entity-pairs linked by INLINEFORM2 in the KB and use them as features to train a binary classifier to predict whether a query INLINEFORM3 should be in the KB  Here, a path between two entities is a sequence of relations linking them  In our work, we adopt the latest PR method, C-PR BIBREF16 and extend it to make it work in the open-world setting  C-PR enumerates paths by performing bidirectional random walks over the KB graph while leveraging the context of the source-target entity-pair  We also adopt and extend the compositional vector space model BIBREF20 , BIBREF21 with continual learning capability for prediction 
Given an OKBC query ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ) (e g , “(Obama, CitizenOf, USA), which means whether Obama a citizen of USA), LiLi interacts with the user (if needed) by dynamically formulating questions (see the interaction example in Figure 1, which will be further explained in §3) and leverages the interactively acquired knowledge (supporting facts (SFs) in the figure) for continued inference  To do so, LiLi formulates a query-specific inference strategy and executes it  We design LiLi in a Reinforcement Learning (RL) setting that performs sub-tasks like formulating and executing strategy, training a prediction model for inference, and knowledge retention for future use  To the best of our knowledge, our work is the first to address the OKBC problem and to propose an interactive learning mechanism to solve it in a continuous or lifelong manner  We empirically verify the effectiveness of LiLi on two standard real-world KBs: Freebase and WordNet  Experimental results show that LiLi is highly effective in terms of its predictive performance and strategy formulation ability 
Related Work
To the best of our knowledge, we are not aware of any knowledge learning system that can learn new knowledge in the conversation process  This section thus discusses other related work 
Among existing KB completion approaches, BIBREF20 extended the vector space model for zero-shot KB inference  However, the model cannot handle unknown entities and can only work on fixed set of unknown relations with known embeddings  Recently, BIBREF22 proposed a method using external text corpus to perform inference on unknown entities  However, the method cannot handle unknown relations  Thus, these methods are not suitable for our open-world setting  None of the existing KB inference methods perform interactive knowledge learning like LiLi  NELL BIBREF23 continuously updates its KB using facts extracted from the Web  Our task is very different as we do not do Web fact extraction (which is also useful)  We focus on user interactions in this paper  Our work is related to interactive language learning (ILL) BIBREF24 , BIBREF25 , but these are not about KB completion  The work in BIBREF26 allows a learner to ask questions in dialogue  However, this work used RL to learn about whether to ask the user or not  The “what to ask aspect" was manually designed by modeling synthetic tasks  LiLi formulates query-specific inference strategies which embed interaction behaviors  Also, no existing dialogue systems BIBREF4 , BIBREF27 , BIBREF28 , BIBREF29 , BIBREF30 employ lifelong learning to train prediction models by using information/knowledge retained in the past 
Our work is related to general lifelong learning in BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36   However, they learn only one type of tasks, e g , supervised, topic modeling or reinforcement learning (RL) tasks  None of them is suitable for our setting, which involves interleaving of RL, supervised and interactive learning  More details about lifelong learning can be found in the book BIBREF31  
Interactive Knowledge Learning (LiLi)
We design LiLi as a combination of two interconnected models: (1) a RL model that learns to formulate a query-specific inference strategy for performing the OKBC task, and (2) a lifelong prediction model to predict whether a triple should be in the KB, which is invoked by an action while executing the inference strategy and is learned for each relation as in C-PR  The framework improves its performance over time through user interaction and knowledge retention  Compared to the existing KB inference methods, LiLi overcomes the following three challenges for OKBC:
1  Mapping open-world to close-world  Being a closed-world method, C-PR cannot extract path features and learn a prediction model when any of INLINEFORM0 , INLINEFORM1 or INLINEFORM2 is unknown  LiLi solves this problem through interactive knowledge acquisition  If INLINEFORM3 is unknown, LiLi asks the user to provide a clue (an example of INLINEFORM4 )  And if INLINEFORM5 or INLINEFORM6 is unknown, LiLi asks the user to provide a link (relation) to connect the unknown entity with an existing entity (automatically selected) in the KB  We refer to such a query as a connecting link query (CLQ)  The acquired knowledge reduces OKBC to KBC and makes the inference task feasible 
2  Spareseness of KB  A main issue of all PR methods like C-PR is the connectivity of the KB graph  If there is no path connecting INLINEFORM0 and INLINEFORM1 in the graph, path enumeration of C-PR gets stuck and inference becomes infeasible  In such cases, LiLi uses a template relation (“@-?-@") as the missing link marker to connect entity-pairs and continues feature extraction  A path containing “@-?-@" is called an incomplete path  Thus, the extracted feature set contains both complete (no missing link) and incomplete paths  Next, LiLi selects an incomplete path from the feature set and asks the user to provide a link for path completion  We refer to such a query as missing link query (MLQ) 
3  Limitation in user knowledge  If the user is unable to respond to MLQs or CLQs, LiLi uses a guessing mechanism (discussed later) to fill the gap  This enables LiLi to continue its inference even if the user cannot answer a system question 
Components of LiLi
As lifelong learning needs to retain knowledge learned from past tasks and use it to help future learning BIBREF31 , LiLi uses a Knowledge Store (KS) for knowledge retention  KS has four components: (i) Knowledge Graph ( INLINEFORM0 ): INLINEFORM1 (the KB) is initialized with base KB triples (see §4) and gets updated over time with the acquired knowledge  (ii) Relation-Entity Matrix ( INLINEFORM2 ): INLINEFORM3 is a sparse matrix, with rows as relations and columns as entity-pairs and is used by the prediction model  Given a triple ( INLINEFORM4 , INLINEFORM5 , INLINEFORM6 ) INLINEFORM7 , we set INLINEFORM8 [ INLINEFORM9 , ( INLINEFORM10 , INLINEFORM11 )] = 1 indicating INLINEFORM12 occurs for pair ( INLINEFORM13 , INLINEFORM14 )  (iii) Task Experience Store ( INLINEFORM15 ): INLINEFORM16 stores the predictive performance of LiLi on past learned tasks in terms of Matthews correlation coefficient (MCC) that measures the quality of binary classification  So, for two tasks INLINEFORM17 and INLINEFORM18 (each relation is a task), if INLINEFORM19 [ INLINEFORM20 ] INLINEFORM21 INLINEFORM22 [ INLINEFORM23 ] [where INLINEFORM24 [ INLINEFORM25 ]=MCC( INLINEFORM26 )], we say C-PR has learned INLINEFORM27 well compared to INLINEFORM28   (iv) Incomplete Feature DB ( INLINEFORM29 ): INLINEFORM30 stores the frequency of an incomplete path INLINEFORM31 in the form of a tuple ( INLINEFORM32 , INLINEFORM33 , INLINEFORM34 ) and is used in formulating MLQs  INLINEFORM35 [( INLINEFORM36 , INLINEFORM37 , INLINEFORM38 )] = INLINEFORM39 implies LiLi has extracted incomplete path INLINEFORM40 INLINEFORM41 times involving entity-pair INLINEFORM42 [( INLINEFORM43 , INLINEFORM44 )] for query relation INLINEFORM45  
The RL model learns even after training whenever it encounters an unseen state (in testing) and thus, gets updated over time  KS is updated continuously over time as a result of the execution of LiLi and takes part in future learning  The prediction model uses lifelong learning (LL), where we transfer knowledge (parameter values) from the model for a past most similar task to help learn for the current task  Similar tasks are identified by factorizing INLINEFORM0 and computing a task similarity matrix INLINEFORM1   Besides LL, LiLi uses INLINEFORM2 to identify poorly learned past tasks and acquire more clues for them to improve its skillset over time 
LiLi also uses a stack, called Inference Stack ( INLINEFORM0 ) to hold query and its state information for RL  LiLi always processes stack top ( INLINEFORM1 [top])  The clues from the user get stored in INLINEFORM2 on top of the query during strategy execution and processed first  Thus, the prediction model for INLINEFORM3 is learned before performing inference on query, transforming OKBC to a KBC problem  Table 1 shows the parameters of LiLi used in the following sections 
Working of LiLi
Given an OKBC query ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ), we represent it as a data instance INLINEFORM3   INLINEFORM4 consists of INLINEFORM5 (the query triple), INLINEFORM6 (interaction limit set for INLINEFORM7 ), INLINEFORM8 (experience list storing the transition history of MDP for INLINEFORM9 in RL) and INLINEFORM10 (mode of INLINEFORM11 ) denoting if INLINEFORM12 is ` INLINEFORM13 ' (training), ` INLINEFORM14 ' (validation), ` INLINEFORM15 ' (evaluation) or ` INLINEFORM16 ' (clue) instance and INLINEFORM17 (feature set)  We denote INLINEFORM18 ( INLINEFORM19 ) as the set of all complete (incomplete) path features in INLINEFORM20   Given a data instance INLINEFORM21 , LiLi starts its initialization as follows: it sets the state as INLINEFORM22 (based on INLINEFORM23 , explained later), pushes the query tuple ( INLINEFORM24 , INLINEFORM25 ) into INLINEFORM26 and feeds INLINEFORM27 [top] to the RL-model for strategy formulation from INLINEFORM28  
Inference Strategy Formulation  We view solving the strategy formulation problem as learning to play an inference game, where the goal is to formulate a strategy that "makes the inference task possible"  Considering PR methods, inference is possible, iff (1) INLINEFORM0 becomes known to its KB (by acquiring clues when INLINEFORM1 is unknown) and (2) path features are extracted between INLINEFORM2 and INLINEFORM3 (which inturn requires INLINEFORM4 and INLINEFORM5 to be known to KB)  If these conditions are met at the end of an episode (when strategy formulation finishes for a given query) of the game, LiLi wins and thus, it trains the prediction model for INLINEFORM6 and uses it for inference 
LiLi's strategy formulation is modeled as a Markov Decision Process (MDP) with finite state ( INLINEFORM0 ) and action ( INLINEFORM1 ) spaces  A state INLINEFORM2 consists of 10 binary state variables (Table 2), each of which keeps track of results of an action INLINEFORM3 taken by LiLi and thus, records the progress in inference process made so far  INLINEFORM4 is the initial state with all state bits set as 0  If the data instance (query) is a clue [ INLINEFORM5 ], INLINEFORM6 [CLUE] is set as 1  INLINEFORM7 consists of 6 actions (Table 3)  INLINEFORM8 , INLINEFORM9 , INLINEFORM10 are processing actions and INLINEFORM11 , INLINEFORM12 , INLINEFORM13 are interactive actions  Whenever INLINEFORM14 is executed, the MDP reaches the terminal state  Given an action INLINEFORM15 in state INLINEFORM16 , if INLINEFORM17 is invalid in INLINEFORM21 or the objective of INLINEFORM22 is unsatisfied (* marked the condition in INLINEFORM23 ), RL receives a negative reward (empirically set); else receives a positive reward   We use Q-learning BIBREF38 with INLINEFORM24 -greedy strategy to learn the optimal policy for training the RL model  Note that, the inference strategy is independent of KB type and correctness of prediction  Thus, the RL-model is trained only once from scratch (reused thereafter for other KBs) and also, independently of the prediction model 
Sometimes the training dataset may not be enough to learn optimal policy for all INLINEFORM0   Thus, encountering an unseen state during test can make RL-model clueless about the action  Given a state INLINEFORM1 , whenever an invalid INLINEFORM2 is chosen, LiLi remains in INLINEFORM3   For INLINEFORM4 , LiLi remains in INLINEFORM5 untill INLINEFORM6 (see Table 1 for INLINEFORM7 )  So, if the state remains the same for ( INLINEFORM8 +1) times, it implies LiLi has encountered a fault (an unseen state)  RL-model instantly switches to the training mode and randomly explores INLINEFORM9 to learn the optimal action (fault-tolerant learning)  While exploring INLINEFORM10 , the model chooses INLINEFORM11 only when it has tried all other INLINEFORM12 to avoid abrupt end of episode 
Execution of Actions  At any given point in time, let ( INLINEFORM0 , INLINEFORM1 ) be the current INLINEFORM2 [top], INLINEFORM3 is the chosen action and the current version of KS components are INLINEFORM4 , INLINEFORM5 , INLINEFORM6 and INLINEFORM7   Then, if INLINEFORM8 is invalid in INLINEFORM9 , LiLi only updates INLINEFORM10 [top] with ( INLINEFORM11 , INLINEFORM12 ) and returns INLINEFORM13 [top] to RL-model  In this process, LiLi adds experience ( INLINEFORM14 , INLINEFORM15 , INLINEFORM16 , INLINEFORM17 ) in INLINEFORM18 and then, replaces INLINEFORM19 [top] with ( INLINEFORM20 , INLINEFORM21 )  If INLINEFORM22 is valid in INLINEFORM23 , LiLi first sets the next state INLINEFORM24 and performs a sequence of operations INLINEFORM25 based on INLINEFORM26 (discussed below)  Unless specified, in INLINEFORM27 , LiLi always monitors INLINEFORM28 and if INLINEFORM29 becomes 0, LiLi sets INLINEFORM30   Also, whenever LiLi asks the user a query, INLINEFORM31 is decremented by 1  Once INLINEFORM32 ends, LiLi updates INLINEFORM33 [top] with ( INLINEFORM34 , INLINEFORM35 ) and returns INLINEFORM36 [top] to RL-model for choosing the next action 
In INLINEFORM0 , LiLi searches INLINEFORM1 , INLINEFORM2 , INLINEFORM3 in INLINEFORM4 and sets appropriate bits in INLINEFORM5 (see Table 2)  If INLINEFORM6 was unknown before and is just added to INLINEFORM7 or is in the bottom INLINEFORM8 % (see Table 1 for INLINEFORM9 ) of INLINEFORM10 , LiLi randomly sets INLINEFORM14 with probability INLINEFORM15   If INLINEFORM16 is a clue and INLINEFORM17 , LiLi updates KS with triple INLINEFORM18 , where ( INLINEFORM19 , INLINEFORM20 , INLINEFORM21 ) and ( INLINEFORM22 , INLINEFORM23 , INLINEFORM24 ) gets added to INLINEFORM25 and INLINEFORM26 , INLINEFORM27 are set as 1 
In INLINEFORM0 , LiLi asks the user to provide a clue (+ve instance) for INLINEFORM1 and corrupts INLINEFORM2 and INLINEFORM3 of the clue once at a time, to generate -ve instances by sampling nodes from INLINEFORM4   These instances help in training prediction model for INLINEFORM5 while executing INLINEFORM6  
In INLINEFORM0 , LiLi selects an incomplete path INLINEFORM1 from INLINEFORM2 to formulate MLQ, such that INLINEFORM3 is most frequently observed for INLINEFORM4 and INLINEFORM5 is high, given by INLINEFORM6   Here, INLINEFORM7 denotes the contextual similarity BIBREF16 of entity-pair INLINEFORM8   If INLINEFORM9 is high, INLINEFORM10 is more likely to possess a relation between them and so, is a good candidate for formulating MLQ  When the user does not respond to MLQ (or CLQ in INLINEFORM11 ), the guessing mechanism is used, which works as follows: Since contextual similarity of entity-pairs is highly correlated with their class labels BIBREF16 , LiLi divides the similarity range [-1, 1] into three segments, using a low ( INLINEFORM12 ) and high ( INLINEFORM13 ) similarity threshold and replaces the missing link with INLINEFORM14 in INLINEFORM15 to make it complete as follows: If INLINEFORM16 , INLINEFORM17 = “@-LooselyRelatedTo-@"; else if INLINEFORM18 , INLINEFORM19 =“@-NotRelatedTo-@"; Otherwise, INLINEFORM20 =“@-RelatedTo-@" 
In INLINEFORM0 , LiLi asks CLQs for connecting unknown entities INLINEFORM1 and/or INLINEFORM2 with INLINEFORM3 by selecting the most contextually relevant node (wrt INLINEFORM4 , INLINEFORM5 ) from INLINEFORM6 , given by link INLINEFORM7   We adopt the contextual relevance idea in BIBREF16 which is computed using word embedding BIBREF39
In INLINEFORM0 , LiLi extracts path features INLINEFORM1 between ( INLINEFORM2 , INLINEFORM3 ) and updates INLINEFORM4 with incomplete features from INLINEFORM5   LiLi always trains the prediction model with complete features INLINEFORM6 and once INLINEFORM7 or INLINEFORM8 , LiLi stops asking MLQs  Thus, in both INLINEFORM9 and INLINEFORM10 , LiLi always monitors INLINEFORM11 to check for the said requirements and sets INLINEFORM12 to control interactions 
In INLINEFORM0 , if LiLi wins the episode, it adds INLINEFORM1 in one of data buffers INLINEFORM2 based on its mode INLINEFORM3   E g , if INLINEFORM4 or INLINEFORM5 , INLINEFORM6 is used for training and added to INLINEFORM7   Similarly validation buffer INLINEFORM8 and evaluation buffer INLINEFORM9 are populated  If INLINEFORM10 , LiLi invokes the prediction model for INLINEFORM11  
Lifelong Relation Prediction  Given a relation INLINEFORM0 , LiLi uses INLINEFORM1 and INLINEFORM2 (see INLINEFORM3 ) to train a prediction model (say, INLINEFORM4 ) with parameters INLINEFORM5   For a unknown INLINEFORM6 , the clue instances get stored in INLINEFORM7 and INLINEFORM8   Thus, LiLi populates INLINEFORM9 by taking 10% (see §4) of the instances from INLINEFORM10 and starts the training  For INLINEFORM11 , LiLi uses a LSTM BIBREF40 to compose the vector representation of each feature INLINEFORM12 as INLINEFORM13 and vector representation of INLINEFORM14 as INLINEFORM15   Next, LiLi computes the prediction value, INLINEFORM16 as sigmoid of the mean cosine similarity of all features and INLINEFORM17 , given by INLINEFORM18 ) and maximize the log-likelihood of INLINEFORM19 for training  Once INLINEFORM20 is trained, LiLi updates INLINEFORM21 [ INLINEFORM22 ] using INLINEFORM23   We also train an inverse model for INLINEFORM24 , INLINEFORM25 by reversing the path features in INLINEFORM26 and INLINEFORM27 which help in lifelong learning (discussed below)  Unlike BIBREF20 , BIBREF21 , while predicting the label for INLINEFORM28 , we compute a relation-specific prediction threshold INLINEFORM29 corresponding to INLINEFORM30 using INLINEFORM31 as: INLINEFORM32 and infer INLINEFORM33 as +ve if INLINEFORM34 and -ve otherwise  Here, INLINEFORM35 ( INLINEFORM36 ) is the mean prediction value for all +ve (-ve) examples in INLINEFORM37  
Models trained on a few examples (e g , clues acquired for unknown INLINEFORM0 ) with randomly initialized weights often perform poorly due to underfitting  Thus, we transfer knowledge (weights) from the past most similar (wrt INLINEFORM1 ) task in a lifelong learning manner BIBREF31   LiLi uses INLINEFORM2 to find the past most similar task for INLINEFORM3 as follows: LiLi computes trancated SVD of INLINEFORM4 as INLINEFORM5 and then, the similarity matrix INLINEFORM6   INLINEFORM7 provides the similarity between relations INLINEFORM8 and INLINEFORM9 in INLINEFORM10   Thus, LiLi chooses a source relation INLINEFORM11 to transfer weights  Here, INLINEFORM12 is the set of all INLINEFORM13 and INLINEFORM14 for which LiLi has already learned a prediction model  Now, if INLINEFORM15 or INLINEFORM16 , LiLi randomly initializes the weights INLINEFORM17 for INLINEFORM18 and proceeds with the training  Otherwise, LiLi uses INLINEFORM19 as initial weights and fine-tunes INLINEFORM20 with a low learning rate 
A Running Example  Considering the example shown in Figure 1, LiLi works as follows: first, LiLi executes INLINEFORM0 and detects that the source entity “Obama" and query relation “CitizenOf" are unknown  Thus, LiLi executes INLINEFORM1 to acquire clue (SF1) for “CitizenOf" and pushes the clue (+ve example) and two generated -ve examples into INLINEFORM2   Once the clues are processed and a prediction model is trained for “CitizenOf" by formulating separate strategies for them, LiLi becomes aware of “CitizenOf"  Now, as the clues have already been popped from INLINEFORM3 , the query becomes INLINEFORM4 and the strategy formulation process for the query resumes  Next, LiLi asks user to provide a connecting link for “Obama" by performing INLINEFORM5   Now, the query entities and relation being known, LiLi enumerates paths between “Obama" and “USA" by performing INLINEFORM6   Let an extracted path be “ INLINEFORM7 " with missing link between ( INLINEFORM8 , INLINEFORM9 )  LiLi asks the user to fill the link by performing INLINEFORM10 and then, extracts the complete feature “ INLINEFORM11 "  The feature set is then fed to the prediction model and inference is made as a result of INLINEFORM12   Thus, the formulated inference strategy is: “ INLINEFORM13 " 
Experiments
We now evaluate LiLi in terms of its predictive performance and strategy formulation abilities 
Data: We use two standard datasets (see Table 4): (1) Freebase FB15k, and (2) WordNet INLINEFORM0   Using each dataset, we build a fairly large graph and use it as the original KB ( INLINEFORM1 ) for evaluation  We also augment INLINEFORM2 with inverse triples ( INLINEFORM3 , INLINEFORM4 , INLINEFORM5 ) for each ( INLINEFORM6 , INLINEFORM7 , INLINEFORM8 ) following existing KBC methods 
Parameter Settings  Unless specified, the empirically set parameters (see Table 1) of LiLi are: INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 , INLINEFORM8 , INLINEFORM9 , INLINEFORM10   For training RL-model with INLINEFORM11 -greedy strategy, we use INLINEFORM12 , INLINEFORM13 , pre-training steps=50000  We used Keras deep learning library to implement and train the prediction model  We set batch-size as 128, max  training epoch as 150, dropout as 0 2, hidden units and embedding size as 300 and learning rate as 5e-3 which is reduced gradually on plateau with factor 0 5 and patience 5  Adam optimizer and early stopping were used in training  We also shuffle INLINEFORM14 in each epoch and adjust class weights inversely proportional to class frequencies in INLINEFORM15  
Labeled Dataset Generation and Simulated User Creation  We create a simulated user for each KB to evaluate LiLi  We create the labeled datasets, the simulated user’s knowledge base ( INLINEFORM0 ), and the base KB ( INLINEFORM1 ) from INLINEFORM2   INLINEFORM3 used as the initial KB graph ( INLINEFORM4 ) of LiLi 
We followed BIBREF16 for labeled dataset generation  For Freebase, we found 86 relations with INLINEFORM0 triples and randomly selected 50 from various domains  We randomly shuffle the list of 50 relations, select 25% of them as unknown relations and consider the rest (75%) as known relations  For each known relation INLINEFORM1 , we randomly shuffle the list of distinct triples for INLINEFORM2 , choose 1000 triples and split them into 60% training, 10% validation and 20% test  Rest 10% along with the leftover (not included in the list of 1000) triples are added to INLINEFORM3   For each unknown relation INLINEFORM4 , we remove all triples of INLINEFORM5 from INLINEFORM6 and add them to INLINEFORM7   In this process, we also randomly choose 20% triples as test instances for unknown INLINEFORM8 which are excluded from INLINEFORM9   Note that, now INLINEFORM10 has at least 10% of chosen triples for each INLINEFORM11 (known and unknown) and so, user is always able to provide clues for both cases  For each labeled dataset, we randomly choose 10% of the entities present in dataset triples, remove triples involving those entities from INLINEFORM12 and add to INLINEFORM13   At this point, INLINEFORM14 gets reduced to INLINEFORM15 and is used as INLINEFORM16 for LiLi  The dataset stats in Table 4 shows that the base KB (60% triples of INLINEFORM17 ) is highly sparse (compared to original KB) which makes the inference task much harder  WordNet dataset being small, we select all 18 relations for evaluation and create labeled dataset, INLINEFORM18 and INLINEFORM19 following Freebase  Although the user may provide clues 100% of the time, it often cannot respond to MLQs and CLQs (due to lack of required triples/facts)  Thus, we further enrich INLINEFORM20 with external KB triples 
Given a relation INLINEFORM0 and an observed triple ( INLINEFORM1 , INLINEFORM2 , INLINEFORM3 ) in training or testing, the pair ( INLINEFORM4 , INLINEFORM5 ) is regarded as a +ve instance for INLINEFORM6   Following BIBREF18 , for each +ve instance ( INLINEFORM7 , INLINEFORM8 ), we generate two negative ones, one by randomly corrupting the source INLINEFORM9 , and the other by corrupting the target INLINEFORM10   Note that, the test triples are not in INLINEFORM11 or INLINEFORM12 and none of the -ve instances overlap with the +ve ones 
Baselines  As none of the existing KBC methods can solve the OKBC problem, we choose various versions of LiLi as baselines 
Single: Version of LiLi where we train a single prediction model INLINEFORM0 for all test relations 
Sep: We do not transfer (past learned) weights for initializing INLINEFORM0 , i e , we disable LL 
F-th): Here, we use a fixed prediction threshold 0 5 instead of relation-specific threshold INLINEFORM0  
BG: The missing or connecting links (when the user does not respond) are filled with “@-RelatedTo-@" blindly, no guessing mechanism 
w/o PTS: LiLi does not ask for additional clues via past task selection for skillset improvement 
Evaluation Metrics  To evaluate the strategy formulation ability, we introduce a measure called Coverage( INLINEFORM0 ), defined as the fraction of total query data instances, for which LiLi has successfully formulated strategies that lead to winning  If LiLi wins on all episodes for a given dataset, INLINEFORM1 is 1 0  To evaluate the predictive performance, we use Avg  MCC and avg  +ve F1 score 
Results and Analysis
Evaluation-I: Strategy Formulation Ability  Table 5 shows the list of inference strategies formulated by LiLi for various INLINEFORM0 and INLINEFORM1 , which control the strategy formulation of LiLi  When INLINEFORM2 , LiLi cannot interact with user and works like a closed-world method  Thus, INLINEFORM3 drops significantly (0 47)  When INLINEFORM4 , i e  with only one interaction per query, LiLi acquires knowledge well for instances where either of the entities or relation is unknown  However, as one unknown entity may appear in multiple test triples, once the entity becomes known, LiLi doesn’t need to ask for it again and can perform inference on future triples causing significant increase in INLINEFORM5 (0 97)  When INLINEFORM6 , LiLi is able to perform inference on all instances and INLINEFORM7 becomes 1  For INLINEFORM8 , LiLi uses INLINEFORM9 only once (as only one MLQ satisfies INLINEFORM10 ) compared to INLINEFORM11   In summary, LiLi’s RL-model can effectively formulate query-specific inference strategies (based on specified parameter values)  Evaluation-II: Predictive Performance  Table 6 shows the comparative performance of LiLi with baselines  To judge the overall improvements, we performed paired t-test considering +ve F1 scores on each relation as paired data  Considering both KBs and all relation types, LiLi outperforms Sep with INLINEFORM12   If we set INLINEFORM13 (training with very few clues), LiLi outperforms Sep with INLINEFORM14 on Freebase considering MCC  Thus, the lifelong learning mechanism is effective in transferring helpful knowledge  Single model performs better than Sep for unknown relations due to the sharing of knowledge (weights) across tasks  However, for known relations, performance drops because, as a new relation arrives to the system, old weights get corrupted and catastrophic forgetting occurs  For unknown relations, as the relations are evaluated just after training, there is no chance for catastrophic forgetting  The performance improvement ( INLINEFORM15 ) of LiLi over F-th on Freebase signifies that the relation-specific threshold INLINEFORM16 works better than fixed threshold 0 5 because, if all prediction values for test instances lie above (or below) 0 5, F-th predicts all instances as +ve (-ve) which degrades its performance  Due to the utilization of contextual similarity (highly correlated with class labels) of entity-pairs, LiLi’s guessing mechanism works better ( INLINEFORM17 ) than blind guessing (BG)  The past task selection mechanism of LiLi also improves its performance over w/o PTS, as it acquires more clues during testing for poorly performed tasks (evaluated on validation set)  For Freebase, due to a large number of past tasks [9 (25% of 38)], the performance difference is more significant ( INLINEFORM18 )  For WordNet, the number is relatively small [3 (25% of 14)] and hence, the difference is not significant 
Evaluation-III: User Interaction vs  Performance  Table 7 shows the results of LiLi by varying clue acquisition rate ( INLINEFORM0 )  We use Freebase for tuning INLINEFORM1 due to its higher number of unknown test relations compared to WordNet  LiLi’s performance improves significantly as it acquires more clues from the user  The results on INLINEFORM2 outperforms ( INLINEFORM3 ) that on INLINEFORM4   Table 8 shows the results of LiLi on user responses to MLQ’s and CLQ’s  Answering MLQ’s and CLQ’s is very hard for simulated users (unlike crowd-sourcing) as often INLINEFORM5 lacks the required triple  Thus, we attempt to analyze how the performance is effected if the user does not respond at all  The results show a clear trend in overall performance improvement when the user responds  However, the improvement is not significant as the simulated user’s query satisfaction rate (1% MLQs and 10% CLQs) is very small  But, the analysis shows the effectiveness of LiLi’s guessing mechanism and continual learning ability that help in achieving avg  +ve F1 of 0 57 and 0 62 on FB and WN respectively with minimal participation of the user 
Conclusion
In this paper, we are interested in building a generic engine for continuous knowledge learning in human-machine conversations  We first showed that the problem underlying the engine can be formulated as an open-world knowledge base completion (OKBC) problem  We then proposed an lifelong interactive learning and inference (LiLi) approach to solving the OKBC problem  OKBC is a generalization of KBC  LiLi solves the OKBC problem by first formulating a query-specific inference strategy using RL and then executing it to solve the problem by interacting with the user in a lifelong learning manner  Experimental results showed the effectiveness of LiLi in terms of both predictive quality and strategy formulation ability  We believe that a system with the LiLi approach can serve as a knowledge learning engine for conversations  Our future work will improve LiLi to make more accurate 
Acknowledgments
This work was supported in part by National Science Foundation (NSF) under grant no  IIS-1407927 and IIS-1650900, and a gift from Huawei Technologies Co Ltd 
Context 3: Introduction
During the first two decades of the 21st century, the sharing and processing of vast amounts of data has become pervasive  This expansion of data sharing and processing capabilities is both a blessing and a curse  Data helps build better information systems for the digital era and enables further research for advanced data management that benefits the society in general  But the use of this very data containing sensitive information conflicts with private data protection, both from an ethical and a legal perspective 
There are several application domains on which this situation is particularly acute  This is the case of the medical domain BIBREF0  There are plenty of potential applications for advanced medical data management that can only be researched and developed using real data; yet, the use of medical data is severely limited –when not entirely prohibited– due to data privacy protection policies 
One way of circumventing this problem is to anonymise the data by removing, replacing or obfuscating the personal information mentioned, as exemplified in Table TABREF1  This task can be done by hand, having people read and anonymise the documents one by one  Despite being a reliable and simple solution, this approach is tedious, expensive, time consuming and difficult to scale to the potentially thousands or millions of documents that need to be anonymised 
For this reason, numerous of systems and approaches have been developed during the last decades to attempt to automate the anonymisation of sensitive content, starting with the automatic detection and classification of sensitive information  Some of these systems rely on rules, patterns and dictionaries, while others use more advanced techniques related to machine learning and, more recently, deep learning 
Given that this paper is concerned with text documents (e g  medical records), the involved techniques are related to Natural Language Processing (NLP)  When using NLP approaches, it is common to pose the problem of document anonymisation as a sequence labelling problem, i e  classifying each token within a sequence as being sensitive information or not  Further, depending on the objective of the anonymisation task, it is also important to determine the type of sensitive information (names of individuals, addresses, age, sex, etc ) 
The anonymisation systems based on NLP techniques perform reasonably well, but are far from perfect  Depending on the difficulty posed by each dataset or the amount of available data for training machine learning models, the performance achieved by these methods is not enough to fully rely on them in certain situations BIBREF0  However, in the last two years, the NLP community has reached an important milestone thanks to the appearance of the so-called Transformers neural network architectures BIBREF1  In this paper, we conduct several experiments in sensitive information detection and classification on Spanish clinical text using BERT (from `Bidirectional Encoder Representations from Transformers') BIBREF2 as the base for a sequence labelling approach  The experiments are carried out on two datasets: the MEDDOCAN: Medical Document Anonymization shared task dataset BIBREF3, and NUBes BIBREF4, a corpus of real medical reports in Spanish  In these experiments, we compare the performance of BERT with other machine-learning-based systems, some of which use language-specific features  Our aim is to evaluate how good a BERT-based model performs without language nor domain specialisation apart from the training data labelled for the task at hand 
The rest of the paper is structured as follows: the next section describes related work about data anonymisation in general and clinical data anonymisation in particular; it also provides a more detailed explanation and background about the Transformers architecture and BERT  Section SECREF3 describes the data involved in the experiments and the systems evaluated in this paper, including the BERT-based system; finally, it details the experimental design  Section SECREF4 introduces the results for each set of experiments  Finally, Section SECREF5 contains the conclusions and future lines of work 
Related Work
The state of the art in the field of Natural Language Processing (NLP) has reached an important milestone in the last couple of years thanks to deep-learning architectures, increasing in several points the performance of new models for almost any text processing task 
The major change started with the Transformers model proposed by vaswani2017attention  It substituted the widely used recurrent and convolutional neural network architectures by another approach based solely on self-attention, obtaining an impressive performance gain  The original proposal was focused on an encoder-decoder architecture for machine translation, but soon the use of Transformers was made more general BIBREF1  There are several other popular models that use Transformers, such as Open AI's GPT and GPT2 BIBREF5, RoBERTa BIBREF6 and the most recent XLNet BIBREF7; still, BERT BIBREF2 is one of the most widespread Transformer-based models 
BERT trains its unsupervised language model using a Masked Language Model and Next Sentence Prediction  A common problem in NLP is the lack of enough training data  BERT can be pre-trained to learn general or specific language models using very large amounts of unlabelled text (e g  web content, Wikipedia, etc ), and this knowledge can be transferred to a different downstream task in a process that receives the name fine-tuning 
devlin2018bert have used fine-tuning to achieve state-of-the-art results on a wide variety of challenging natural language tasks, such as text classification, Question Answering (QA) and Named Entity Recognition and Classification (NERC)  BERT has also been used successfully by other community practitioners for a wide range of NLP-related tasks BIBREF8, BIBREF9 
Regarding the task of data anonymisation in particular, anonymisation systems may follow different approaches and pursue different objectives (Cormode and Srivastava, 2009)  The first objective of these systems is to detect and classify the sensitive information contained in the documents to be anonymised  In order to achieve that, they use rule-based approaches, Machine Learning (ML) approaches, or a combination of both 
Although most of these efforts are for English texts –see, among others, the i2b2 de-identification challenges BIBREF10, BIBREF11, dernon2016deep, or khin2018deep–, other languages are also attracting growing interest  Some examples are mamede2016automated for Portuguese and tveit2004anonymization for Norwegian  With respect to the anonymisation of text written in Spanish, recent studies include medina2018building, hassan2018anonimizacion and garcia2018automating  Most notably, in 2019 the first community challenge about anonymisation of medical documents in Spanish, MEDDOCAN BIBREF3, was held as part of the IberLEF initiative  The winners of the challenge –the Neither-Language-nor-Domain-Experts (NLNDE) BIBREF12– achieved F1-scores as high as 0 975 in the task of sensitive information detection and categorisation by using recurrent neural networks with Conditional Random Field (CRF) output layers 
At the same challenge, mao2019hadoken occupied the 8th position among 18 participants using BERT  According to the description of the system, the authors used BERT-Base Multilingual Cased and an output CRF layer  However, their system is $\sim $3 F1-score points below our implementation without the CRF layer 
Materials and Methods
The aim of this paper is to evaluate BERT's multilingual model and compare it to other established machine-learning algorithms in a specific task: sensitive data detection and classification in Spanish clinical free text  This section describes the data involved in the experiments and the systems evaluated  Finally, we introduce the experimental setup 
Materials and Methods ::: Data
Two datasets are exploited in this article  Both datasets consist of plain text containing clinical narrative written in Spanish, and their respective manual annotations of sensitive information in BRAT BIBREF13 standoff format  In order to feed the data to the different algorithms presented in Section SECREF7, these datasets were transformed to comply with the commonly used BIO sequence representation scheme BIBREF14 
Materials and Methods ::: Data ::: NUBes-PHI
NUBes BIBREF4 is a corpus of around 7,000 real medical reports written in Spanish and annotated with negation and uncertainty information  Before being published, sensitive information had to be manually annotated and replaced for the corpus to be safely shared  In this article, we work with the NUBes version prior to its anonymisation, that is, with the manual annotations of sensitive information  It follows that the version we work with is not publicly available and, due to contractual restrictions, we cannot reveal the provenance of the data  In order to avoid confusion between the two corpus versions, we henceforth refer to the version relevant in this paper as NUBes-PHI (from `NUBes with Personal Health Information') 
NUBes-PHI consists of 32,055 sentences annotated for 11 different sensitive information categories  Overall, it contains 7,818 annotations  The corpus has been randomly split into train (72%), development (8%) and test (20%) sets to conduct the experiments described in this paper  The size of each split and the distribution of the annotations can be consulted in Tables and , respectively 
The majority of sensitive information in NUBes-PHI are temporal expressions (`Date' and `Time'), followed by healthcare facility mentions (`Hospital'), and the age of the patient  Mentions of people are not that frequent, with physician names (`Doctor') occurring much more often than patient names (`Patient')  The least frequent sensitive information types, which account for $\sim $10% of the remaining annotations, consist of the patient's sex, job, and kinship, and locations other than healthcare facilities (`Location')  Finally, the tag `Other' includes, for instance, mentions to institutions unrelated to healthcare and whether the patient is right- or left-handed  It occurs just 36 times

ground_truth:  the same datasets as BIBREF7

same datasets as BIBREF7

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What metrics are used to establish that this makes chatbots more knowledgeable and better at learning and conversation? 
Context 3: 
Materials and Methods ::: Data ::: The MEDDOCAN corpus
The organisers of the MEDDOCAN shared task BIBREF3 curated a synthetic corpus of clinical cases enriched with sensitive information by health documentalists  In this regard, the MEDDOCAN evaluation scenario could be said to be somewhat far from the real use case the technology developed for the shared task is supposed to be applied in  However, at the moment it also provides the only public means for a rigorous comparison between systems for sensitive health information detection in Spanish texts 
The size of the MEDDOCAN corpus is shown in Table   Compared to NUBes-PHI (Table ), this corpus contains more sensitive information annotations, both in absolute and relative terms 
The sensitive annotation categories considered in MEDDOCAN differ in part from those in NUBes-PHI  Most notably, it contains finer-grained labels for location-related mentions –namely, `Address', `Territory', and `Country'–, and other sensitive information categories that we did not encounter in NUBes-PHI (e g , identifiers, phone numbers, e-mail addresses, etc )  In total, the MEDDOCAN corpus has 21 sensitive information categories  We refer the reader to the organisers' article BIBREF3 for more detailed information about this corpus 
Materials and Methods ::: Systems
Apart from experimenting with a pre-trained BERT model, we have run experiments with other systems and baselines, to compare them and obtain a better perspective about BERT's performance in these datasets 
Materials and Methods ::: Systems ::: Baseline
As the simplest baseline, a sensitive data recogniser and classifier has been developed that consists of regular-expressions and dictionary look-ups  For each category to detect a specific method has been implemented  For instance, the Date, Age, Time and Doctor detectors are based on regular-expressions; Hospital, Sex, Kinship, Location, Patient and Job are looked up in dictionaries  The dictionaries are hand-crafted from the training data available, except for the Patient's case, for which the possible candidates considered are the 100 most common female and male names in Spain according to the Instituto Nacional de Estadística (INE; Spanish Statistical Office) 
Materials and Methods ::: Systems ::: CRF
Conditional Random Fields (CRF) BIBREF15 have been extensively used for tasks of sequential nature  In this paper, we propose as one of the competitive baselines a CRF classifier trained with sklearn-crfsuite for Python 3 5 and the following configuration: algorithm = lbfgs; maximum iterations = 100; c1 = c2 = 0 1; all transitions = true; optimise = false  The features extracted from each token are as follows:
[noitemsep]
prefixes and suffixes of 2 and 3 characters;
the length of the token in characters and the length of the sentence in tokens;
whether the token is all-letters, a number, or a sequence of punctuation marks;
whether the token contains the character `@';
whether the token is the start or end of the sentence;
the token's casing and the ratio of uppercase characters, digits, and punctuation marks to its length;
and, the lemma, part-of-speech tag, and named-entity tag given by ixa-pipes BIBREF16 upon analysing the sentence the token belongs to 
Noticeably, none of the features used to train the CRF classifier is domain-dependent  However, the latter group of features is language dependent 
Materials and Methods ::: Systems ::: spaCy
spaCy is a widely used NLP library that implements state-of-the-art text processing pipelines, including a sequence-labelling pipeline similar to the one described by strubell2017fast  spaCy offers several pre-trained models in Spanish, which perform basic NLP tasks such as Named Entity Recognition (NER)  In this paper, we have trained a new NER model to detect NUBes-PHI labels  For this purpose, the new model uses all the labels of the training corpus coded with its context at sentence level  The network optimisation parameters and dropout values are the ones recommended in the documentation for small datasets  Finally, the model is trained using batches of size 64  No more features are included, so the classifier is language-dependent but not domain-dependent 
Materials and Methods ::: Systems ::: BERT
As introduced earlier, BERT has shown an outstanding performance in NERC-like tasks, improving the start-of-the-art results for almost every dataset and language  We take the same approach here, by using the model BERT-Base Multilingual Cased with a Fully Connected (FC) layer on top to perform a fine-tuning of the whole model for an anonymisation task in Spanish clinical data  Our implementation is built on PyTorch and the PyTorch-Transformers library BIBREF1  The training phase consists in the following steps (roughly depicted in Figure ):
Pre-processing: since we are relying on a pre-trained BERT model, we must match the same configuration by using a specific tokenisation and vocabulary  BERT also needs that the inputs contains special tokens to signal the beginning and the end of each sequence 
Fine-tuning: the pre-processed sequence is fed into the model  BERT outputs the contextual embeddings that encode each of the inputted tokens  This embedding representation for each token is fed into the FC linear layer after a dropout layer (with a 0 1 dropout probability), which in turn outputs the logits for each possible class  The cross-entropy loss function is calculated comparing the logits and the gold labels, and the error is back-propagated to adjust the model parameters 
We have trained the model using an AdamW optimiser BIBREF17 with the learning rate set to 3e-5, as recommended by devlin2018bert, and with a gradient clipping of 1 0  We also applied a learning-rate scheduler that warms up the learning rate from zero to its maximum value as the training progresses, which is also a common practice  For each experiment set proposed below, the training was run with an early-stopping patience of 15 epochs  Then, the model that performed best against the development set was used to produce the reported results 
The experiments were run on a 64-core server with operating system Ubuntu 16 04, 250GB of RAM memory, and 4 GeForce RTX 2080 GPUs with 11GB of memory  The maximum sequence length was set at 500 and the batch size at 12  In this setting, each epoch –a full pass through all the training data– required about 10 minutes to complete 
Materials and Methods ::: Experimental design
We have conducted experiments with BERT in the two datasets of Spanish clinical narrative presented in Section SECREF3 The first experiment set uses NUBes-PHI, a corpus of real medical reports manually annotated with sensitive information  Because this corpus is not publicly available, and in order to compare the BERT-based model to other related published systems, the second set of experiments uses the MEDDOCAN 2019 shared task competition dataset  The following sections provide greater detail about the two experimental setups 
Materials and Methods ::: Experimental design ::: Experiment A: NUBes-PHI
In this experiment set, we evaluate all the systems presented in Section SECREF7, namely, the rule-based baseline, the CRF classifier, the spaCy entity tagger, and BERT  The evaluation comprises three scenarios of increasing difficulty:
[noitemsep]
- Evaluates the performance of the systems at predicting whether each token is sensitive or non-sensitive; that is, the measurements only take into account whether a sensitive token has been recognised or not, regardless of the BIO label and the category assigned  This scenario shows how good a system would be at obfuscating sensitive data (e g , by replacing sensitive tokens with asterisks) 
- We measure the performance of the systems at predicting the sensitive information type of each token –i e , the 11 categories presented in Section SECREF5 or `out'  Detecting entity types correctly is important if a system is going to be used to replace sensitive data by fake data of the same type (e g , random people names) 
- This is the strictest evaluation, as it takes into account both the BIO label and the category assigned to each individual token  Being able to discern between two contiguous sensitive entities of the same type is relevant not only because it is helpful when producing fake replacements, but because it also yields more accurate statistics of the sensitive information present in a given document collection 
The systems are evaluated in terms of micro-average precision, recall and F1-score in all the scenarios 
In addition to the scenarios proposed, a subject worth being studied is the need of labelled data  Manually labelled data is an scarce and expensive resource, which for some application domains or languages is difficult to come by  In order to obtain an estimation of the dependency of each system on the available amount of training data, we have retrained all the compared models using decreasing amounts of data –from 100% of the available training instances to just 1%  The same data subsets have been used to train all the systems  Due to the knowledge transferred from the pre-trained BERT model, the BERT-based model is expected to be more robust to data scarcity than those that start their training from scratch 
Materials and Methods ::: Experimental design ::: Experiment B: MEDDOCAN
In this experiment set, our BERT implementation is compared to several systems that participated in the MEDDOCAN challenge: a CRF classifier BIBREF18, a spaCy entity recogniser BIBREF18, and NLNDE BIBREF12, the winner of the shared task and current state of the art for sensitive information detection and classification in Spanish clinical text  Specifically, we include the results of a domain-independent NLNDE model (S2), and the results of a model enriched with domain-specific embeddings (S3)  Finally, we include the results obtained by mao2019hadoken with a CRF output layer on top of BERT embeddings  MEDDOCAN consists of two scenarios:
[noitemsep]
- This evaluation measures how good a system is at detecting sensitive text spans, regardless of the category assigned to them

ground_truth: Coverage, Avg. MCC and avg. +ve F1 score

strategy formulation ability, we introduce a measure called Coverage( INLINEFORM0 ), To evaluate the predictive performance, we use Avg. MCC and avg. +ve F1 score

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the clinical datasets used in the paper?
Context 3: 
- In this scenario, systems are required to match exactly not only the boundaries of each sensitive span, but also the category assigned 
The systems are evaluated in terms of micro-averaged precision, recall and F-1 score  Note that, in contrast to the evaluation in Experiment A, MEDDOCAN measurements are entity-based instead of tokenwise  An exhaustive explanation of the MEDDOCAN evaluation procedure is available online, as well as the official evaluation script, which we used to obtain the reported results 
Results
This section describes the results obtained in the two sets of experiments: NUBes-PHI and MEDDOCAN 
Results ::: Experiment A: NUBes-PHI
Table shows the results of the conducted experiments in NUBes-PHI for all the compared systems  The included baseline serves to give a quick insight about how challenging the data is  With simple regular expressions and gazetteers a precision of 0 853 is obtained  On the other hand, the recall, which directly depends on the coverage provided by the rules and resources, drops to 0 469  Hence, this task is unlikely to be solved without the generalisation capabilities provided by machine-learning and deep-learning models 
Regarding the detection scenario –that is, the scenario concerned with a binary classification to determine whether each individual token conveys sensitive information or not–, it can be observed that BERT outperforms its competitors  A fact worth highlighting is that, according to these results, BERT achieves a precision lower than the rest of the systems (i e , it makes more false positive predictions); in exchange, it obtains a remarkably higher recall  Noticeably, it reaches a recall of 0 979, improving by more than 4 points the second-best system, spaCy 
The table also shows the results for the relaxed metric that only takes into account the entity type detected, regardless of the BIO label (i e , ignoring whether the token is at the beginning or in the middle of a sensitive sequence of tokens)  The conclusions are very similar to those extracted previously, with BERT gaining 2 1 points of F1-score over the CRF based approach  The confusion matrices of the predictions made by CRF, spaCy, and BERT in this scenario are shown in Table   As can bee seen, BERT has less difficulty in predicting correctly less frequent categories, such as `Location', `Job', and `Patient'  One of the most common mistakes according to the confusion matrices is classifying hospital names as `Location' instead of the more accurate `Hospital'; this is hardly a harmful error, given that a hospital is actually a location  Last, the category `Other' is completely leaked by all the compared systems, most likely due to its almost total lack of support in both training and evaluation datasets 
To finish with this experiment set, Table also shows the strict classification precision, recall and F1-score for the compared systems  Despite the fact that, in general, the systems obtain high values, BERT outperforms them again  BERT's F1-score is 1 9 points higher than the next most competitive result in the comparison  More remarkably, the recall obtained by BERT is about 5 points above 
Upon manual inspection of the errors committed by the BERT-based model, we discovered that it has a slight tendency towards producing ill-formed BIO sequences (e g, starting a sensitive span with `Inside' instead of `Begin'; see Table )  We could expect that complementing the BERT-based model with a CRF layer on top would help enforce the emission of valid sequences, alleviating this kind of errors and further improving its results 
Finally, Figure shows the impact of decreasing the amount of training data in the detection scenario  It shows the difference in precision, recall, and F1-score with respect to that obtained using 100% of the training data  A general downward trend can be observed, as one would expect: less training data leads to less accurate predictions  However, the BERT-based model is the most robust to training-data reduction, showing an steadily low performance loss  With 1% of the dataset (230 training instances), the BERT-based model only suffers a striking 7-point F1-score loss, in contrast to the 32 and 39 points lost by the CRF and spaCy models, respectively  This steep performance drop stems to a larger extent from recall decline, which is not that marked in the case of BERT  Overall, these results indicate that the transfer-learning achieved through the BERT multilingual pre-trained model not only helps obtain better results, but also lowers the need of manually labelled data for this application domain 
Results ::: Experiment B: MEDDOCAN
The results of the two MEDDOCAN scenarios –detection and classification– are shown in Table   These results follow the same pattern as in the previous experiments, with the CRF classifier being the most precise of all, and BERT outperforming both the CRF and spaCy classifiers thanks to its greater recall  We also show the results of mao2019hadoken who, despite of having used a BERT-based system, achieve lower scores than our models  The reason why it should be so remain unclear 
With regard to the winner of the MEDDOCAN shared task, the BERT-based model has not improved the scores obtained by neither the domain-dependent (S3) nor the domain-independent (S2) NLNDE model  However, attending to the obtained results, BERT remains only 0 3 F1-score points behind, and would have achieved the second position among all the MEDDOCAN shared task competitors  Taking into account that only 3% of the gold labels remain incorrectly annotated, the task can be considered almost solved, and it is not clear if the differences among the systems are actually significant, or whether they stem from minor variations in initialisation or a long-tail of minor labelling inconsistencies 
Conclusions and Future Work
In this work we have briefly introduced the problems related to data privacy protection in clinical domain  We have also described some of the groundbreaking advances on the Natural Language Processing field due to the appearance of Transformers-based deep-learning architectures and transfer learning from very large general-domain multilingual corpora, focusing our attention in one of its most representative examples, Google's BERT model 
In order to assess the performance of BERT for Spanish clinical data anonymisation, we have conducted several experiments with a BERT-based sequence labelling approach using the pre-trained multilingual BERT model shared by Google as the starting point for the model training  We have compared this BERT-based sequence labelling against other methods and systems  One of the experiments uses the MEDDOCAN 2019 shared task dataset, while the other uses a novel Spanish clinical reports dataset called NUBes-PHI 
The results of the experiments show that, in NUBes-PHI, the BERT-based model outperforms the other systems without requiring any adaptation or domain-specific feature engineering, just by being trained on the provided labelled data  Interestingly, the BERT-based model obtains a remarkably higher recall than the other systems  High recall is a desirable outcome because, when anonymising sensible documents, the accidental leak of sensible data is likely to be more dangerous than the unintended over-obfuscation of non-sensitive text 
Further, we have conducted an additional experiment on this dataset by progressively reducing the training data for all the compared systems  The BERT-based model shows the highest robustness to training-data scarcity, loosing only 7 points of F1-score when trained on 230 instances instead of 21,371  These observation are in line with the results obtained by the NLP community using BERT for other tasks 
The experiments with the MEDDOCAN 2019 shared task dataset follow the same pattern  In this case, the BERT-based model falls 0 3 F1-score points behind the shared task winning system, but it would have achieved the second position in the competition with no further refinement 
Since we have used a pre-trained multilingual BERT model, the same approach is likely to work for other languages just by providing some labelled training data  Further, this is the simplest fine-tuning that can be performed based on BERT  More sophisticated fine-tuning layers could help improve the results  For example, it could be expected that a CRF layer helped enforce better BIO tagging sequence predictions  Precisely, mao2019hadoken participated in the MEDDOCAN competition using a BERT+CRF architecture, but their reported scores are about 3 points lower than our implementation  From the description of their work, it is unclear what the source of this score difference could be 
Further, at the time of writing this paper, new multilingual pre-trained models and Transformer architectures have become available  It would not come as a surprise that these new resources and systems –e g , XLM-RoBERTa BIBREF19 or BETO BIBREF20, a BERT model fully pre-trained on Spanish texts– further advanced the state of the art in this task 
Acknowledgements
This work has been supported by Vicomtech and partially funded by the project DeepReading (RTI2018-096846-B-C21, MCIU/AEI/FEDER,UE) 

ground_truth: MEDDOCAN, NUBes-PHI

MEDDOCAN, NUBes