prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What baseline model is used?
Context 1: Introduction
Wikipedia is the largest source of open and collaboratively curated knowledge in the world  Introduced in 2001, it has evolved into a reference work with around 5m pages for the English Wikipedia alone  In addition, entities and event pages are updated quickly via collaborative editing and all edits are encouraged to include source citations, creating a knowledge base which aims at being both timely as well as authoritative  As a result, it has become the preferred source of information consumption about entities and events  Moreso, this knowledge is harvested and utilized in building knowledge bases like YAGO BIBREF0 and DBpedia BIBREF1 , and used in applications like text categorization BIBREF2 , entity disambiguation BIBREF3 , entity ranking BIBREF4 and distant supervision BIBREF5 , BIBREF6  
However, not all Wikipedia pages referring to entities (entity pages) are comprehensive: relevant information can either be missing or added with a delay  Consider the city of New Orleans and the state of Odisha which were severely affected by cyclones Hurricane Katrina and Odisha Cyclone, respectively  While Katrina finds extensive mention in the entity page for New Orleans, Odisha Cyclone which has 5 times more human casualties (cf  Figure FIGREF2 ) is not mentioned in the page for Odisha  Arguably Katrina and New Orleans are more popular entities, but Odisha Cyclone was also reported extensively in national and international news outlets  This highlights the lack of important facts in trunk and long-tail entity pages, even in the presence of relevant sources  In addition, previous studies have shown that there is an inherent delay or lag when facts are added to entity pages BIBREF7  
To remedy these problems, it is important to identify information sources that contain novel and salient facts to a given entity page  However, not all information sources are equal  The online presence of major news outlets is an authoritative source due to active editorial control and their articles are also a timely container of facts  In addition, their use is in line with current Wikipedia editing practice, as is shown in BIBREF7 that almost 20% of current citations in all entity pages are news articles  We therefore propose news suggestion as a novel task that enhances entity pages and reduces delay while keeping its pages authoritative 
Existing efforts to populate Wikipedia BIBREF8 start from an entity page and then generate candidate documents about this entity using an external search engine (and then post-process them)  However, such an approach lacks in (a) reproducibility since rankings vary with time with obvious bias to recent news (b) maintainability since document acquisition for each entity has to be periodically performed  To this effect, our news suggestion considers a news article as input, and determines if it is valuable for Wikipedia  Specifically, given an input news article INLINEFORM0 and a state of Wikipedia, the news suggestion problem identifies the entities mentioned in INLINEFORM1 whose entity pages can improve upon suggesting INLINEFORM2   Most of the works on knowledge base acceleration BIBREF9 , BIBREF10 , BIBREF11 , or Wikipedia page generation BIBREF8 rely on high quality input sources which are then utilized to extract textual facts for Wikipedia page population  In this work, we do not suggest snippets or paraphrases but rather entire articles which have a high potential importance for entity pages  These suggested news articles could be consequently used for extraction, summarization or population either manually or automatically – all of which rely on high quality and relevant input sources 
We identify four properties of good news recommendations: salience, relative authority, novelty and placement  First, we need to identify the most salient entities in a news article  This is done to avoid pollution of entity pages with only marginally related news  Second, we need to determine whether the news is important to the entity as only the most relevant news should be added to a precise reference work  To do this, we compute the relative authority of all entities in the news article: we call an entity more authoritative than another if it is more popular or noteworthy in the real world  Entities with very high authority have many news items associated with them and only the most relevant of these should be included in Wikipedia whereas for entities of lower authority the threshold for inclusion of a news article will be lower  Third, a good recommendation should be able to identify novel news by minimizing redundancy coming from multiple news articles  Finally, addition of facts is facilitated if the recommendations are fine-grained, i e , recommendations are made on the section level rather than the page level (placement) 
Approach and Contributions  We propose a two-stage news suggestion approach to entity pages  In the first stage, we determine whether a news article should be suggested for an entity, based on the entity's salience in the news article, its relative authority and the novelty of the article to the entity page  The second stage takes into account the class of the entity for which the news is suggested and constructs section templates from entities of the same class  The generation of such templates has the advantage of suggesting and expanding entity pages that do not have a complete section structure in Wikipedia, explicitly addressing long-tail and trunk entities  Afterwards, based on the constructed template our method determines the best fit for the news article with one of the sections 
We evaluate the proposed approach on a news corpus consisting of 351,982 articles crawled from the news external references in Wikipedia from 73,734 entity pages  Given the Wikipedia snapshot at a given year (in our case [2009-2014]), we suggest news articles that might be cited in the coming years  The existing news references in the entity pages along with their reference date act as our ground-truth to evaluate our approach  In summary, we make the following contributions 
Related Work
As we suggest a new problem there is no current work addressing exactly the same task  However, our task has similarities to Wikipedia page generation and knowledge base acceleration  In addition, we take inspiration from Natural Language Processing (NLP) methods for salience detection 
Wikipedia Page Generation is the problem of populating Wikipedia pages with content coming from external sources  Sauper and Barzilay BIBREF8 propose an approach for automatically generating whole entity pages for specific entity classes  The approach is trained on already-populated entity pages of a given class (e g  `Diseases') by learning templates about the entity page structure (e g  diseases have a treatment section)  For a new entity page, first, they extract documents via Web search using the entity title and the section title as a query, for example `Lung Cancer'+`Treatment'  As already discussed in the introduction, this has problems with reproducibility and maintainability  However, their main focus is on identifying the best paragraphs extracted from the collected documents  They rank the paragraphs via an optimized supervised perceptron model for finding the most representative paragraph that is the least similar to paragraphs in other sections  This paragraph is then included in the newly generated entity page  Taneva and Weikum BIBREF12 propose an approach that constructs short summaries for the long tail  The summaries are called `gems' and the size of a `gem' can be user defined  They focus on generating summaries that are novel and diverse  However, they do not consider any structure of entities, which is present in Wikipedia 
In contrast to BIBREF8 and BIBREF12 , we actually focus on suggesting entire documents to Wikipedia entity pages  These are authoritative documents (news), which are highly relevant for the entity, novel for the entity and in which the entity is salient  Whereas relevance in Sauper and Barzilay is implicitly computed by web page ranking we solve that problem by looking at relative authority and salience of an entity, using the news article and entity page only  As Sauper and Barzilay concentrate on empty entity pages, the problem of novelty of their content is not an issue in their work whereas it is in our case which focuses more on updating entities  Updating entities will be more and more important the bigger an existing reference work is  Both the approaches in BIBREF8 and BIBREF12 (finding paragraphs and summarization) could then be used to process the documents we suggest further  Our concentration on news is also novel 
Knowledge Base Acceleration  In this task, given specific information extraction templates, a given corpus is analyzed in order to find worthwhile mentions of an entity or snippets that match the templates  Balog BIBREF9 , BIBREF10 recommend news citations for an entity  Prior to that, the news articles are classified for their appropriateness for an entity, where as features for the classification task they use entity, document, entity-document and temporal features  The best performing features are those that measure similarity between an entity and the news document  West et al  BIBREF13 consider the problem of knowledge base completion, through question answering and complete missing facts in Freebase based on templates, i e  Frank_Zappa bornIn Baltymore, Maryland 
In contrast, we do not extract facts for pre-defined templates but rather suggest news articles based on their relevance to an entity  In cases of long-tail entities, we can suggest to add a novel section through our abstraction and generation of section templates at entity class level 
Entity Salience  Determining which entities are prominent or salient in a given text has a long history in NLP, sparked by the linguistic theory of Centering BIBREF14   Salience has been used in pronoun and co-reference resolution BIBREF15 , or to predict which entities will be included in an abstract of an article BIBREF11   Frequent features to measure salience include the frequency of an entity in a document, positioning of an entity, grammatical function or internal entity structure (POS tags, head nouns etc )  These approaches are not currently aimed at knowledge base generation or Wikipedia coverage extension but we postulate that an entity's salience in a news article is a prerequisite to the news article being relevant enough to be included in an entity page  We therefore use the salience features in BIBREF11 as part of our model  However, these features are document-internal — we will show that they are not sufficient to predict news inclusion into an entity page and add features of entity authority, news authority and novelty that measure the relations between several entities, between entity and news article as well as between several competing news articles 
Terminology and Problem Definition
We are interested in named entities mentioned in documents  An entity INLINEFORM0 can be identified by a canonical name, and can be mentioned differently in text via different surface forms  We canonicalize these mentions to entity pages in Wikipedia, a method typically known as entity linking  We denote the set of canonicalized entities extracted and linked from a news article INLINEFORM1 as INLINEFORM2   For example, in Figure FIGREF7 , entities are canonicalized into Wikipedia entity pages (e g  Odisha is canonicalized to the corresponding article)  For a collection of news articles INLINEFORM3 , we further denote the resulting set of entities by INLINEFORM4  
Information in an entity page is organized into sections and evolves with time as more content is added  We refer to the state of Wikipedia at a time INLINEFORM0 as INLINEFORM1 and the set of sections for an entity page INLINEFORM2 as its entity profile INLINEFORM3   Unlike news articles, text in Wikipedia could be explicitly linked to entity pages through anchors  The set of entities explicitly referred in text from section INLINEFORM4 is defined as INLINEFORM5   Furthermore, Wikipedia induces a category structure over its entities, which is exploited by knowledge bases like YAGO (e g  Barack_Obama isA Person)  Consequently, each entity page belongs to one or more entity categories or classes INLINEFORM6   Now we can define our news suggestion problem below:
Definition 1 (News Suggestion Problem) Given a set of news articles INLINEFORM0 and set of Wikipedia entity pages INLINEFORM1 (from INLINEFORM2 ) we intend to suggest a news article INLINEFORM3 published at time INLINEFORM4 to entity page INLINEFORM5 and additionally to the most relevant section for the entity page INLINEFORM6  
Approach Overview
We approach the news suggestion problem by decomposing it into two tasks:
AEP: Article–Entity placement
ASP: Article–Section placement
In this first step, for a given entity-news pair INLINEFORM0 , we determine whether the given news article INLINEFORM1 should be suggested (we will refer to this as `relevant') to entity INLINEFORM2   To generate such INLINEFORM3 pairs, we perform the entity linking process, INLINEFORM4 , for INLINEFORM5  
The article–entity placement task (described in detail in Section SECREF16 ) for a pair INLINEFORM0 outputs a binary label (either `non-relevant' or `relevant') and is formalized in Equation EQREF14   DISPLAYFORM0
In the second step, we take into account all `relevant' pairs INLINEFORM0 and find the correct section for article INLINEFORM1 in entity INLINEFORM2 , respectively its profile INLINEFORM3 (see Section SECREF30 )  The article–section placement task, determines the correct section for the triple INLINEFORM4 , and is formalized in Equation EQREF15   DISPLAYFORM0
In the subsequent sections we describe in details how we approach the two tasks for suggesting news articles to entity pages 
News Article Suggestion
In this section, we provide an overview of the news suggestion approach to Wikipedia entity pages (see Figure FIGREF7 )  The approach is split into two tasks: (i) article-entity (AEP) and (ii) article-section (ASP) placement  For a Wikipedia snapshot INLINEFORM0 and a news corpus INLINEFORM1 , we first determine which news articles should be suggested to an entity INLINEFORM2   We will denote our approach for AEP by INLINEFORM3   Finally, we determine the most appropriate section for the ASP task and we denote our approach with INLINEFORM4  
In the following, we describe the process of learning the functions INLINEFORM0 and INLINEFORM1   We introduce features for the learning process, which encode information regarding the entity salience, relative authority and novelty in the case of AEP task  For the ASP task, we measure the overall fit of an article to the entity sections, with the entity being an input from AEP task  Additionally, considering that the entity profiles INLINEFORM2 are incomplete, in the case of a missing section we suggest and expand the entity profiles based on section templates generated from entities of the same class INLINEFORM3 (see Section UID34 ) 
Article–Entity Placement
In this step we learn the function INLINEFORM0 to correctly determine whether INLINEFORM1 should be suggested for INLINEFORM2 , basically a binary classification model (0=`non-relevant' and 1=`relevant')  Note that we are mainly interested in finding the relevant pairs in this task  For every news article, the number of disambiguated entities is around 30 (but INLINEFORM3 is suggested for only two of them on average)  Therefore, the distribution of `non-relevant' and `relevant' pairs is skewed towards the earlier, and by simply choosing the `non-relevant' label we can achieve a high accuracy for INLINEFORM4   Finding the relevant pairs is therefore a considerable challenge 
An article INLINEFORM0 is suggested to INLINEFORM1 by our function INLINEFORM2 if it fulfills the following properties  The entity INLINEFORM3 is salient in INLINEFORM4 (a central concept), therefore ensuring that INLINEFORM5 is about INLINEFORM6 and that INLINEFORM7 is important for INLINEFORM8   Next, given the fact there might be many articles in which INLINEFORM9 is salient, we also look at the reverse property, namely whether INLINEFORM10 is important for INLINEFORM11   We do this by comparing the authority of INLINEFORM12 (which is a measure of popularity of an entity, such as its frequency of mention in a whole corpus) with the authority of its co-occurring entities in INLINEFORM13 , leading to a feature we call relative authority  The intuition is that for an entity that has overall lower authority than its co-occurring entities, a news article is more easily of importance  Finally, if the article we are about to suggest is already covered in the entity profile INLINEFORM14 , we do not wish to suggest redundant information, hence the novelty  Therefore, the learning objective of INLINEFORM15 should fulfill the following properties  Table TABREF21 shows a summary of the computed features for INLINEFORM16  
Salience: entity INLINEFORM0 should be a salient entity in news article INLINEFORM1
Relative Authority: the set of entities INLINEFORM0 with which INLINEFORM1 co-occurs should have higher authority than INLINEFORM2 , making INLINEFORM3 important for INLINEFORM4
Novelty: news article INLINEFORM0 should provide novel information for entity INLINEFORM1 taking into account its profile INLINEFORM2
Baseline Features  As discussed in Section SECREF2 , a variety of features that measure salience of an entity in text are available from the NLP community  We reimplemented the ones in Dunietz and Gillick BIBREF11   This includes a variety of features, e g  positional features, occurrence frequency and the internal POS structure of the entity and the sentence it occurs in  Table 2 in BIBREF11 gives details 
Relative Entity Frequency  Although frequency of mention and positional features play some role in baseline features, their interaction is not modeled by a single feature nor do the positional features encode more than sentence position  We therefore suggest a novel feature called relative entity frequency, INLINEFORM0 , that has three properties : (i) It rewards entities for occurring throughout the text instead of only in some parts of the text, measured by the number of paragraphs it occurs in (ii) it rewards entities that occur more frequently in the opening paragraphs of an article as we model INLINEFORM1 as an exponential decay function  The decay corresponds to the positional index of the news paragraph  This is inspired by the news-specific discourse structure that tends to give short summaries of the most important facts and entities in the opening paragraphs  (iii) it compares entity frequency to the frequency of its co-occurring mentions as the weight of an entity appearing in a specific paragraph, normalized by the sum of the frequencies of other entities in INLINEFORM2   DISPLAYFORM0
where, INLINEFORM0 represents a news paragraph from INLINEFORM1 , and with INLINEFORM2 we indicate the set of all paragraphs in INLINEFORM3   The frequency of INLINEFORM4 in a paragraph INLINEFORM5 is denoted by INLINEFORM6   With INLINEFORM7 and INLINEFORM8 we indicate the number of paragraphs in which entity INLINEFORM9 occurs, and the total number of paragraphs, respectively 
Relative Authority  In this case, we consider the comparative relevance of the news article to the different entities occurring in it  As an example, let us consider the meeting of the Sudanese bishop Elias Taban with Hillary Clinton  Both entities are salient for the meeting  However, in Taban's Wikipedia page, this meeting is discussed prominently with a corresponding news reference, whereas in Hillary Clinton's Wikipedia page it is not reported at all  We believe this is not just an omission in Clinton's page but mirrors the fact that for the lesser known Taban the meeting is big news whereas for the more famous Clinton these kind of meetings are a regular occurrence, not all of which can be reported in what is supposed to be a selection of the most important events for her  Therefore, if two entities co-occur, the news is more relevant for the entity with the lower a priori authority 
The a priori authority of an entity (denoted by INLINEFORM0 ) can be measured in several ways  We opt for two approaches: (i) probability of entity INLINEFORM1 occurring in the corpus INLINEFORM2 , and (ii) authority assessed through centrality measures like PageRank BIBREF16   For the second case we construct the graph INLINEFORM3 consisting of entities in INLINEFORM4 and news articles in INLINEFORM5 as vertices  The edges are established between INLINEFORM6 and entities in INLINEFORM7 , that is INLINEFORM8 , and the out-links from INLINEFORM9 , that is INLINEFORM10 (arrows present the edge direction) 
Starting from a priori authority, we proceed to relative authority by comparing the a priori authority of co-occurring entities in INLINEFORM0   We define the relative authority of INLINEFORM1 as the proportion of co-occurring entities INLINEFORM2 that have a higher a priori authority than INLINEFORM3 (see Equation EQREF28   DISPLAYFORM0
As we might run the danger of not suggesting any news articles for entities with very high a priori authority (such as Clinton) due to the strict inequality constraint, we can relax the constraint such that the authority of co-occurring entities is above a certain threshold 
News Domain Authority  The news domain authority addresses two main aspects  Firstly, if bundled together with the relative authority feature, we can ensure that dependent on the entity authority, we suggest news from authoritative sources, hence ensuring the quality of suggested articles  The second aspect is in a news streaming scenario where multiple news domains report the same event — ideally only articles coming from authoritative sources would fulfill the conditions for the news suggestion task 
The news domain authority is computed based on the number of news references in Wikipedia coming from a particular news domain INLINEFORM0   This represents a simple prior that a news article INLINEFORM1 is from domain INLINEFORM2 in corpus INLINEFORM3   We extract the domains by taking the base URLs from the news article URLs 
An important feature when suggesting an article INLINEFORM0 to an entity INLINEFORM1 is the novelty of INLINEFORM2 w r t the already existing entity profile INLINEFORM3   Studies BIBREF17 have shown that on comparable collections to ours (TREC GOV2) the number of duplicates can go up to INLINEFORM4   This figure is likely higher for major events concerning highly authoritative entities on which all news media will report 
Given an entity INLINEFORM0 and the already added news references INLINEFORM1 up to year INLINEFORM2 , the novelty of INLINEFORM3 at year INLINEFORM4 is measured by the KL divergence between the language model of INLINEFORM5 and articles in INLINEFORM6   We combine this measure with the entity overlap of INLINEFORM7 and INLINEFORM8   The novelty value of INLINEFORM9 is given by the minimal divergence value  Low scores indicate low novelty for the entity profile INLINEFORM10  
N(n|e) = n'Nt-1{DKL((n') || (n)) + DKL((N) || (n)) 
DKL((n') || (n))  (1-) jaccard((n'),(n))} where INLINEFORM0 is the KL divergence of the language models ( INLINEFORM1 and INLINEFORM2 ), whereas INLINEFORM3 is the mixing weight ( INLINEFORM4 ) between the language models INLINEFORM5 and the entity overlap in INLINEFORM6 and INLINEFORM7  
Here we introduce the evaluation setup and analyze the results for the article–entity (AEP) placement task  We only report the evaluation metrics for the `relevant' news-entity pairs  A detailed explanation on why we focus on the `relevant' pairs is provided in Section SECREF16  
Baselines  We consider the following baselines for this task 
B1  The first baseline uses only the salience-based features by Dunietz and Gillick BIBREF11  
B2  The second baseline assigns the value relevant to a pair INLINEFORM0 , if and only if INLINEFORM1 appears in the title of INLINEFORM2  
Learning Models  We use Random Forests (RF) BIBREF23   We learn the RF on all computed features in Table TABREF21   The optimization on RF is done by splitting the feature space into multiple trees that are considered as ensemble classifiers  Consequently, for each classifier it computes the margin function as a measure of the average count of predicting the correct class in contrast to any other class  The higher the margin score the more robust the model 
Metrics  We compute precision P, recall R and F1 score for the relevant class  For example, precision is the number of news-entity pairs we correctly labeled as relevant compared to our ground truth divided by the number of all news-entity pairs we labeled as relevant 
The following results measure the effectiveness of our approach in three main aspects: (i) overall performance of INLINEFORM0 and comparison to baselines, (ii) robustness across the years, and (iii) optimal model for the AEP placement task 
Performance  Figure FIGREF55 shows the results for the years 2009 and 2013, where we optimized the learning objective with instances from year INLINEFORM0 and evaluate on the years INLINEFORM1 (see Section SECREF46 )  The results show the precision–recall curve  The red curve shows baseline B1 BIBREF11 , and the blue one shows the performance of INLINEFORM2   The curve shows for varying confidence scores (high to low) the precision on labeling the pair INLINEFORM3 as `relevant'  In addition, at each confidence score we can compute the corresponding recall for the `relevant' label  For high confidence scores on labeling the news-entity pairs, the baseline B1 achieves on average a precision score of P=0 50, while INLINEFORM4 has P=0 93  We note that with the drop in the confidence score the corresponding precision and recall values drop too, and the overall F1 score for B1 is around F1=0 2, in contrast we achieve an average score of F1=0 67 
It is evident from Figure FIGREF55 that for the years 2009 and 2013, INLINEFORM0 significantly outperforms the baseline B1  We measure the significance through the t-test statistic and get a p-value of INLINEFORM1   The improvement we achieve over B1 in absolute numbers, INLINEFORM2 P=+0 5 in terms of precision for the years between 2009 and 2014, and a similar improvement in terms of F1 score  The improvement for recall is INLINEFORM3 R=+0 4  The relative improvement over B1 for P and F1 is almost 1 8 times better, while for recall we are 3 5 times better  In Table TABREF58 we show the overall scores for the evaluation metrics for B1 and INLINEFORM4   Finally, for B2 we achieve much poorer performance, with average scores of P=0 21, R=0 20 and F1=0 21 
Robustness  In Table TABREF58 , we show the overall performance for the years between 2009 and 2013  An interesting observation we make is that we have a very robust performance and the results are stable across the years  If we consider the experimental setup, where for year INLINEFORM0 we optimize the learning objective with only 74k training instances and evaluate on the rest of the instances, it achieves a very good performance  We predict with F1=0 68 the remaining 469k instances for the years INLINEFORM1  
The results are particularly promising considering the fact that the distribution between our two classes is highly skewed  On average the number of `relevant' pairs account for only around INLINEFORM0 of all pairs  A good indicator to support such a statement is the kappa (denoted by INLINEFORM1 ) statistic  INLINEFORM2 measures agreement between the algorithm and the gold standard on both labels while correcting for chance agreement (often expected due to extreme distributions)  The INLINEFORM3 scores for B1 across the years is on average INLINEFORM4 , while for INLINEFORM5 we achieve a score of INLINEFORM6 (the maximum score for INLINEFORM7 is 1) 
In Figure FIGREF60 we show the impact of the individual feature groups that contribute to the superior performance in comparison to the baselines  Relative entity frequency from the salience feature, models the entity salience as an exponentially decaying function based on the positional index of the paragraph where the entity appears  The performance of INLINEFORM0 with relative entity frequency from the salience feature group is close to that of all the features combined  The authority and novelty features account to a further improvement in terms of precision, by adding roughly a 7%-10% increase  However, if both feature groups are considered separately, they significantly outperform the baseline B1 
Article–Section Placement
We model the ASP placement task as a successor of the AEP task  For all the `relevant' news entity pairs, the task is to determine the correct entity section  Each section in a Wikipedia entity page represents a different topic  For example, Barack Obama has the sections `Early Life', `Presidency', `Family and Personal Life' etc  However, many entity pages have an incomplete section structure  Incomplete or missing sections are due to two Wikipedia properties  First, long-tail entities miss information and sections due to their lack of popularity  Second, for all entities whether popular or not, certain sections might occur for the first time due to real world developments  As an example, the entity Germanwings did not have an `Accidents' section before this year's disaster, which was the first in the history of the airline 
Even if sections are missing for certain entities, similar sections usually occur in other entities of the same class (e g  other airlines had disasters and therefore their pages have an accidents section)  We exploit such homogeneity of section structure and construct templates that we use to expand entity profiles  The learning objective for INLINEFORM0 takes into account the following properties:
Section-templates: account for incomplete section structure for an entity profile INLINEFORM0 by constructing section templates INLINEFORM1 from an entity class INLINEFORM2
Overall fit: measures the overall fit of a news article to sections in the section templates INLINEFORM0
Given the fact that entity profiles are often incomplete, we construct section templates for every entity class  We group entities based on their class INLINEFORM0 and construct section templates INLINEFORM1   For different entity classes, e g  Person and Location, the section structure and the information represented in those section varies heavily  Therefore, the section templates are with respect to the individual classes in our experimental setup (see Figure FIGREF42 )  DISPLAYFORM0
Generating section templates has two main advantages  Firstly, by considering class-based profiles, we can overcome the problem of incomplete individual entity profiles and thereby are able to suggest news articles to sections that do not yet exist in a specific entity INLINEFORM0   The second advantage is that we are able to canonicalize the sections, i e  `Early Life' and `Early Life and Childhood' would be treated similarly 
To generate the section template INLINEFORM0 , we extract all sections from entities of a given type INLINEFORM1 at year INLINEFORM2   Next, we cluster the entity sections, based on an extended version of k–means clustering BIBREF18 , namely x–means clustering introduced in Pelleg et al  which estimates the number of clusters efficiently BIBREF19   As a similarity metric we use the cosine similarity computed based on the tf–idf models of the sections  Using the x–means algorithm we overcome the requirement to provide the number of clusters k beforehand  x–means extends the k–means algorithm, such that a user only specifies a range [ INLINEFORM3 , INLINEFORM4 ] that the number of clusters may reasonably lie in 
The learning objective of INLINEFORM0 is to determine the overall fit of a news article INLINEFORM1 to one of the sections in a given section template INLINEFORM2   The template is pre-determined by the class of the entity for which the news is suggested as relevant by INLINEFORM3   In all cases, we measure how well INLINEFORM4 fits each of the sections INLINEFORM5 as well as the specific entity section INLINEFORM6   The section profiles in INLINEFORM7 represent the aggregated entity profiles from all entities of class INLINEFORM8 at year INLINEFORM9  
To learn INLINEFORM0 we rely on a variety of features that consider several similarity aspects as shown in Table TABREF31   For the sake of simplicity we do not make the distinction in Table TABREF31 between the individual entity section and class-based section similarities, INLINEFORM1 and INLINEFORM2 , respectively  Bear in mind that an entity section INLINEFORM3 might be present at year INLINEFORM4 but not at year INLINEFORM5 (see for more details the discussion on entity profile expansion in Section UID69 ) 
Topic  We use topic similarities to ensure (i) that the content of INLINEFORM0 fits topic-wise with a specific section text and (ii) that it has a similar topic to previously referred news articles in that section  In a pre-processing stage we compute the topic models for the news articles, entity sections INLINEFORM1 and the aggregated class-based sections in INLINEFORM2   The topic models are computed using LDA BIBREF20   We only computed a single topic per article/section as we are only interested in topic term overlaps between article and sections  We distinguish two main features: the first feature measures the overlap of topic terms between INLINEFORM3 and the entity section INLINEFORM4 and INLINEFORM5 , and the second feature measures the overlap of the topic model of INLINEFORM6 against referred news articles in INLINEFORM7 at time INLINEFORM8  
Syntactic  These features represent a mechanism for conveying the importance of a specific text snippet, solely based on the frequency of specific POS tags (i e  NNP, CD etc ), as commonly used in text summarization tasks  Following the same intuition as in BIBREF8 , we weigh the importance of articles by the count of specific POS tags  We expect that for different sections, the importance of POS tags will vary  We measure the similarity of POS tags in a news article against the section text  Additionally, we consider bi-gram and tri-gram POS tag overlap  This exploits similarity in syntactical patterns between the news and section text 
Lexical  As lexical features, we measure the similarity of INLINEFORM0 against the entity section text INLINEFORM1 and the aggregate section text INLINEFORM2   Further, we distinguish between the overall similarity of INLINEFORM3 and that of the different news paragraphs ( INLINEFORM4 which denotes the paragraphs of INLINEFORM5 up to the 5th paragraph)  A higher similarity on the first paragraphs represents a more confident indicator that INLINEFORM6 should be suggested to a specific section INLINEFORM7   We measure the similarity based on two metrics: (i) the KL-divergence between the computed language models and (ii) cosine similarity of the corresponding paragraph text INLINEFORM8 and section text 
Entity-based  Another feature set we consider is the overlap of named entities and their corresponding entity classes  For different entity sections, we expect to find a particular set of entity classes that will correlate with the section, e g  `Early Life' contains mostly entities related to family, school, universities etc 
Frequency  Finally, we gather statistics about the number of entities, paragraphs, news article length, top– INLINEFORM0 entities and entity classes, and the frequency of different POS tags  Here we try to capture patterns of articles that are usually cited in specific sections 
Evaluation Plan
In this section we outline the evaluation plan to verify the effectiveness of our learning approaches  To evaluate the news suggestion problem we are faced with two challenges 
What comprises the ground truth for such a task ?
How do we construct training and test splits given that entity pages consists of text added at different points in time ?
Consider the ground truth challenge  Evaluating if an arbitrary news article should be included in Wikipedia is both subjective and difficult for a human if she is not an expert  An invasive approach, which was proposed by Barzilay and Sauper BIBREF8 , adds content directly to Wikipedia and expects the editors or other users to redact irrelevant content over a period of time  The limitations of such an evaluation technique is that content added to long-tail entities might not be evaluated by informed users or editors in the experiment time frame  It is hard to estimate how much time the added content should be left on the entity page  A more non-invasive approach could involve crowdsourcing of entity and news article pairs in an IR style relevance assessment setup  The problem of such an approach is again finding knowledgeable users or experts for long-tail entities  Thus the notion of relevance of a news recommendation is challenging to evaluate in a crowd setup 
We take a slightly different approach by making an assumption that the news articles already present in Wikipedia entity pages are relevant  To this extent, we extract a dataset comprising of all news articles referenced in entity pages (details in Section SECREF40 )  At the expense of not evaluating the space comprising of news articles absent in Wikipedia, we succeed in (i) avoiding restrictive assumptions about the quality of human judgments, (ii) being invasive and polluting Wikipedia, and (iii) deriving a reusable test bed for quicker experimentation 
The second challenge of construction of training and test set separation is slightly easier and is addressed in Section SECREF46  
Datasets
The datasets we use for our experimental evaluation are directly extracted from the Wikipedia entity pages and their revision history  The generated data represents one of the contributions of our paper  The datasets are the following:
Entity Classes  We focus on a manually predetermined set of entity classes for which we expect to have news coverage  The number of analyzed entity classes is 27, including INLINEFORM0 entities with at least one news reference  The entity classes were selected from the DBpedia class ontology  Figure FIGREF42 shows the number of entities per class for the years (2009-2014) 
News Articles  We extract all news references from the collected Wikipedia entity pages  The extracted news references are associated with the sections in which they appear  In total there were INLINEFORM0 news references, and after crawling we end up with INLINEFORM1 successfully crawled news articles  The details of the news article distribution, and the number of entities and sections from which they are referred are shown in Table TABREF44  
Article-Entity Ground-truth  The dataset comprises of the news and entity pairs INLINEFORM0   News-entity pairs are relevant if the news article is referenced in the entity page  Non-relevant pairs (i e  negative training examples) consist of news articles that contain an entity but are not referenced in that entity's page  If a news article INLINEFORM1 is referred from INLINEFORM2 at year INLINEFORM3 , the features are computed taking into account the entity profiles at year INLINEFORM4  
Article-Section Ground-truth  The dataset consists of the triple INLINEFORM0 , where INLINEFORM1 , where we assume that INLINEFORM2 has already been determined as relevant  We therefore have a multi-class classification problem where we need to determine the section of INLINEFORM3 where INLINEFORM4 is cited  Similar to the article-entity ground truth, here too the features compute the similarity between INLINEFORM5 , INLINEFORM6 and INLINEFORM7  
Data Pre-Processing
We POS-tag the news articles and entity profiles INLINEFORM0 with the Stanford tagger BIBREF21   For entity linking the news articles, we use TagMe! BIBREF22 with a confidence score of 0 3  On a manual inspection of a random sample of 1000 disambiguated entities, the accuracy is above 0 9  On average, the number of entities per news article is approximately 30  For entity linking the entity profiles, we simply follow the anchor text that refers to Wikipedia entities 
Train and Testing Evaluation Setup
We evaluate the generated supervised models for the two tasks, AEP and ASP, by splitting the train and testing instances  It is important to note that for the pairs INLINEFORM0 and the triple INLINEFORM1 , the news article INLINEFORM2 is referenced at time INLINEFORM3 by entity INLINEFORM4 , while the features take into account the entity profile at time INLINEFORM5   This avoids any `overlapping' content between the news article and the entity page, which could affect the learning task of the functions INLINEFORM6 and INLINEFORM7   Table TABREF47 shows the statistics of train and test instances  We learn the functions at year INLINEFORM8 and test on instances for the years greater than INLINEFORM9   Please note that we do not show the performance for year 2014 as we do not have data for 2015 for evaluation 
Article-Section Placement
Here we show the evaluation setup for ASP task and discuss the results with a focus on three main aspects, (i) the overall performance across the years, (ii) the entity class specific performance, and (iii) the impact on entity profile expansion by suggesting missing sections to entities based on the pre-computed templates 
Baselines  To the best of our knowledge, we are not aware of any comparable approach for this task  Therefore, the baselines we consider are the following:
S1: Pick the section from template INLINEFORM0 with the highest lexical similarity to INLINEFORM1 : S1 INLINEFORM2
S2: Place the news into the most frequent section in INLINEFORM0
Learning Models  We use Random Forests (RF) BIBREF23 and Support Vector Machines (SVM) BIBREF24   The models are optimized taking into account the features in Table TABREF31   In contrast to the AEP task, here the scale of the number of instances allows us to learn the SVM models  The SVM model is optimized using the INLINEFORM0 loss function and uses the Gaussian kernels 
Metrics  We compute precision P as the ratio of news for which we pick a section INLINEFORM0 from INLINEFORM1 and INLINEFORM2 conforms to the one in our ground-truth (see Section SECREF40 )  The definition of recall R and F1 score follows from that of precision 
Figure FIGREF66 shows the overall performance and a comparison of our approach (when INLINEFORM0 is optimized using SVM) against the best performing baseline S2  With the increase in the number of training instances for the ASP task the performance is a monotonically non-decreasing function  For the year 2009, we optimize the learning objective of INLINEFORM1 with around 8% of the total instances, and evaluate on the rest  The performance on average is around P=0 66 across all classes  Even though for many classes the performance is already stable (as we will see in the next section), for some classes we improve further  If we take into account the years between 2010 and 2012, we have an increase of INLINEFORM2 P=0 17, with around 70% of instances used for training and the remainder for evaluation  For the remaining years the total improvement is INLINEFORM3 P=0 18 in contrast to the performance at year 2009 
On the other hand, the baseline S1 has an average precision of P=0 12  The performance across the years varies slightly, with the year 2011 having the highest average precision of P=0 13  Always picking the most frequent section as in S2, as shown in Figure FIGREF66 , results in an average precision of P=0 17, with a uniform distribution across the years 
Here we show the performance of INLINEFORM0 decomposed for the different entity classes  Specifically we analyze the 27 classes in Figure FIGREF42   In Table TABREF68 , we show the results for a range of years (we omit showing all years due to space constraints)  For illustration purposes only, we group them into four main classes ( INLINEFORM1 Person, Organization, Location, Event INLINEFORM2 ) and into the specific sub-classes shown in the second column in Table TABREF68   For instance, the entity classes OfficeHolder and Politician are aggregated into Person–Politics 
It is evident that in the first year the performance is lower in contrast to the later years  This is due to the fact that as we proceed, we can better generalize and accurately determine the correct fit of an article INLINEFORM0 into one of the sections from the pre-computed templates INLINEFORM1   The results are already stable for the year range INLINEFORM2   For a few Person sub-classes, e g  Politics, Entertainment, we achieve an F1 score above 0 9  These additionally represent classes with a sufficient number of training instances for the years INLINEFORM3   The lowest F1 score is for the Criminal and Television classes  However, this is directly correlated with the insufficient number of instances 
The baseline approaches for the ASP task perform poorly  S1, based on lexical similarity, has a varying performance for different entity classes  The best performance is achieved for the class Person – Politics, with P=0 43  This highlights the importance of our feature choice and that the ASP cannot be considered as a linear function, where the maximum similarity yields the best results  For different entity classes different features and combination of features is necessary  Considering that S2 is the overall best performing baseline, through our approach INLINEFORM0 we have a significant improvement of over INLINEFORM1 P=+0 64 
The models we learn are very robust and obtain high accuracy, fulfilling our pre-condition for accurate news suggestions into the entity sections  We measure the robustness of INLINEFORM0 through the INLINEFORM1 statistic  In this case, we have a model with roughly 10 labels (corresponding to the number of sections in a template INLINEFORM2 )  The score we achieve shows that our model predicts with high confidence with INLINEFORM3  
The last analysis is the impact we have on expanding entity profiles INLINEFORM0 with new sections  Figure FIGREF70 shows the ratio of sections for which we correctly suggest an article INLINEFORM1 to the right section in the section template INLINEFORM2   The ratio here corresponds to sections that are not present in the entity profile at year INLINEFORM3 , that is INLINEFORM4   However, given the generated templates INLINEFORM5 , we can expand the entity profile INLINEFORM6 with a new section at time INLINEFORM7   In details, in the absence of a section at time INLINEFORM8 , our model trains well on similar sections from the section template INLINEFORM9 , hence we can predict accurately the section and in this case suggest its addition to the entity profile  With time, it is obvious that the expansion rate decreases at later years as the entity profiles become more `complete' 
This is particularly interesting for expanding the entity profiles of long-tail entities as well as updating entities with real-world emerging events that are added constantly  In many cases such missing sections are present at one of the entities of the respective entity class INLINEFORM0   An obvious case is the example taken in Section SECREF16 , where the `Accidents' is rather common for entities of type Airline  However, it is non-existent for some specific entity instances, i e Germanwings airline 
Through our ASP approach INLINEFORM0 , we are able to expand both long-tail and trunk entities  We distinguish between the two types of entities by simply measuring their section text length  The real distribution in the ground truth (see Section SECREF40 ) is 27% and 73% are long-tail and trunk entities, respectively  We are able to expand the entity profiles for both cases and all entity classes without a significant difference, with the only exception being the class Creative Work, where we expand significantly more trunk entities 
Conclusion and Future Work
In this work, we have proposed an automated approach for the novel task of suggesting news articles to Wikipedia entity pages to facilitate Wikipedia updating  The process consists of two stages  In the first stage, article–entity placement, we suggest news articles to entity pages by considering three main factors, such as entity salience in a news article, relative authority and novelty of news articles for an entity page  In the second stage, article–section placement, we determine the best fitting section in an entity page  Here, we remedy the problem of incomplete entity section profiles by constructing section templates for specific entity classes  This allows us to add missing sections to entity pages  We carry out an extensive experimental evaluation on 351,983 news articles and 73,734 entities coming from 27 distinct entity classes  For the first stage, we achieve an overall performance with P=0 93, R=0 514 and F1=0 676, outperforming our baseline competitors significantly  For the second stage, we show that we can learn incrementally to determine the correct section for a news article based on section templates  The overall performance across different classes is P=0 844, R=0 885 and F1=0 860 
In the future, we will enhance our work by extracting facts from the suggested news articles  Results suggest that the news content cited in entity pages comes from the first paragraphs  However, challenging task such as the canonicalization and chronological ordering of facts, still remain 
Context 2: Data
We build and test our MMT models on the Multi30K dataset BIBREF21   Each image in Multi30K contains one English (EN) description taken from Flickr30K BIBREF22 and human translations into German (DE), French (FR) and Czech BIBREF23 , BIBREF24 , BIBREF25   The dataset contains 29,000 instances for training, 1,014 for development, and 1,000 for test  We only experiment with German and French, which are languages for which we have in-house expertise for the type of analysis we present  In addition to the official Multi30K test set (test 2016), we also use the test set from the latest WMT evaluation competition, test 2018 BIBREF25  
Degradation of source
In addition to using the Multi30K dataset as is (standard setup), we probe the ability of our models to address the three linguistic phenomena where additional context has been proved important (Section ): ambiguities, gender-neutral words and noisy input  In a controlled experiment where we aim to remove the influence of frequency biases, we degrade the source sentences by masking words through three strategies to replace words by a placeholder: random source words, ambiguous source words and gender unmarked source words  The procedure is applied to the train, validation and test sets  For the resulting dataset generated for each setting, we compare models having access to text-only context versus additional text and multimodal contexts  We seek to get insights into the contribution of each type of context to address each type of degradation 
In this setting (RND) we simulate erroneous source words by randomly dropping source content words  We first tag the entire source sentences using the spacy toolkit BIBREF26 and then drop nouns, verbs, adjectives and adverbs and replace these with a default BLANK token  By focusing on content words, we differ from previous work that suggests that neural machine translation is robust to non-content word noise in the source BIBREF27  
In this setting (AMB), we rely on the MLT dataset BIBREF11 which provides a list of source words with multiple translations in the Multi30k training set  We replace ambiguous words with the BLANK token in the source language, which results in two language-specific datasets 
In this setting (PERS), we use the Flickr Entities dataset BIBREF28 to identify all the words that were annotated by humans as corresponding to the category person  We then replace such source words with the BLANK token 
The statistics of the resulting datasets for the three degradation strategies are shown in Table TABREF10   We note that RND and PERS are the same for language pairs as the degradation only depends on the source side, while for AMB the words replaced depend on the target language 
Models
Based on the models described in Section we experiment with eight variants: (a) baseline transformer model (base); (b) base with AIC (base+sum); (c) base with AIF using spacial (base+att) or object based (base+obj) image features; (d) standard deliberation model (del); (e) deliberation models enriched with image information: del+sum, del+att and del+obj 
Training
In all cases, we optimise our models with cross entropy loss  For deliberation network models, we first train the standard transformer model until convergence, and use it to initialise the encoder and first-pass decoder  For each of the training samples, we follow BIBREF19 and obtain a set of 10-best samples from the first pass decoder, with a beam search of size 10  We use these as the first-pass decoder samples  We use Adam as optimiser BIBREF29 and train the model until convergence 
Results
In this section we present results of our experiments, first in the original dataset without any source degradation (Section SECREF18 ) and then in the setup with various source degradation strategies (Section SECREF25 ) 
Standard setup
Table TABREF14 shows the results of our main experiments on the 2016 and 2018 test sets for French and German  We use Meteor BIBREF31 as the main metric, as in the WMT tasks BIBREF25   We compare our transformer baseline to transformer models enriched with image information, as well as to the deliberation models, with or without image information 
We first note that our multimodal models achieve the state of the art performance for transformer networks (constrained models) on the English-German dataset, as compared to BIBREF30   Second, our deliberation models lead to significant improvements over this baseline across test sets (average INLINEFORM0 , INLINEFORM1 ) 
Transformer-based models enriched with image information (base+sum, base+att and base+obj), on the other hand, show no major improvements with respect to the base performance  This is also the case for deliberation models with image information (del+sum, del+att, del+obj), which do not show significant improvement over the vanilla deliberation performance (del) 
However, as it has been shown in the WMT shared tasks on MMT BIBREF23 , BIBREF24 , BIBREF25 , automatic metrics often fail to capture nuances in translation quality, such as, the ones we expect the visual modality to help with, which – according to human perception – lead to better translations  To test this assumption in our settings, we performed human evaluation involving professional translators and native speakers of both French and German (three annotators) 
The annotators were asked to rank randomly selected test samples according to how well they convey the meaning of the source, given the image (50 samples per language pair per annotator)  For each source segment, the annotator was shown the outputs of three systems: base+att, the current MMT state-of-the-art BIBREF30 , del and del+obj  A rank could be assigned from 1 to 3, allowing ties BIBREF32   Annotators could assign zero rank to all translations if they were judged incomprehensible  Following the common practice in WMT BIBREF32 , each system was then assigned a score which reflects the proportion of times it was judged to be better or equal other systems 
Table TABREF19 shows the human evaluation results  They are consistent with the automatic evaluation results when it comes to the preference of humans towards the deliberation-based setups, but show a more positive outlook regarding the addition of visual information (del+obj over del) for French 
Manual inspection of translations suggests that deliberation setups tend to improve both the grammaticality and adequacy of the first pass outputs  For German, the most common modifications performed by the second-pass decoder are substitutions of adjectives and verbs (for test 2016, 15% and 12% respectively, of all the edit distance operations)
Context 3: Introduction
During the first two decades of the 21st century, the sharing and processing of vast amounts of data has become pervasive  This expansion of data sharing and processing capabilities is both a blessing and a curse  Data helps build better information systems for the digital era and enables further research for advanced data management that benefits the society in general  But the use of this very data containing sensitive information conflicts with private data protection, both from an ethical and a legal perspective 
There are several application domains on which this situation is particularly acute  This is the case of the medical domain BIBREF0  There are plenty of potential applications for advanced medical data management that can only be researched and developed using real data; yet, the use of medical data is severely limited –when not entirely prohibited– due to data privacy protection policies 
One way of circumventing this problem is to anonymise the data by removing, replacing or obfuscating the personal information mentioned, as exemplified in Table TABREF1  This task can be done by hand, having people read and anonymise the documents one by one  Despite being a reliable and simple solution, this approach is tedious, expensive, time consuming and difficult to scale to the potentially thousands or millions of documents that need to be anonymised 
For this reason, numerous of systems and approaches have been developed during the last decades to attempt to automate the anonymisation of sensitive content, starting with the automatic detection and classification of sensitive information  Some of these systems rely on rules, patterns and dictionaries, while others use more advanced techniques related to machine learning and, more recently, deep learning 
Given that this paper is concerned with text documents (e g  medical records), the involved techniques are related to Natural Language Processing (NLP)  When using NLP approaches, it is common to pose the problem of document anonymisation as a sequence labelling problem, i e  classifying each token within a sequence as being sensitive information or not  Further, depending on the objective of the anonymisation task, it is also important to determine the type of sensitive information (names of individuals, addresses, age, sex, etc ) 
The anonymisation systems based on NLP techniques perform reasonably well, but are far from perfect  Depending on the difficulty posed by each dataset or the amount of available data for training machine learning models, the performance achieved by these methods is not enough to fully rely on them in certain situations BIBREF0  However, in the last two years, the NLP community has reached an important milestone thanks to the appearance of the so-called Transformers neural network architectures BIBREF1  In this paper, we conduct several experiments in sensitive information detection and classification on Spanish clinical text using BERT (from `Bidirectional Encoder Representations from Transformers') BIBREF2 as the base for a sequence labelling approach  The experiments are carried out on two datasets: the MEDDOCAN: Medical Document Anonymization shared task dataset BIBREF3, and NUBes BIBREF4, a corpus of real medical reports in Spanish  In these experiments, we compare the performance of BERT with other machine-learning-based systems, some of which use language-specific features  Our aim is to evaluate how good a BERT-based model performs without language nor domain specialisation apart from the training data labelled for the task at hand 
The rest of the paper is structured as follows: the next section describes related work about data anonymisation in general and clinical data anonymisation in particular; it also provides a more detailed explanation and background about the Transformers architecture and BERT  Section SECREF3 describes the data involved in the experiments and the systems evaluated in this paper, including the BERT-based system; finally, it details the experimental design  Section SECREF4 introduces the results for each set of experiments  Finally, Section SECREF5 contains the conclusions and future lines of work 
Related Work
The state of the art in the field of Natural Language Processing (NLP) has reached an important milestone in the last couple of years thanks to deep-learning architectures, increasing in several points the performance of new models for almost any text processing task 
The major change started with the Transformers model proposed by vaswani2017attention  It substituted the widely used recurrent and convolutional neural network architectures by another approach based solely on self-attention, obtaining an impressive performance gain  The original proposal was focused on an encoder-decoder architecture for machine translation, but soon the use of Transformers was made more general BIBREF1  There are several other popular models that use Transformers, such as Open AI's GPT and GPT2 BIBREF5, RoBERTa BIBREF6 and the most recent XLNet BIBREF7; still, BERT BIBREF2 is one of the most widespread Transformer-based models 
BERT trains its unsupervised language model using a Masked Language Model and Next Sentence Prediction  A common problem in NLP is the lack of enough training data  BERT can be pre-trained to learn general or specific language models using very large amounts of unlabelled text (e g  web content, Wikipedia, etc ), and this knowledge can be transferred to a different downstream task in a process that receives the name fine-tuning 
devlin2018bert have used fine-tuning to achieve state-of-the-art results on a wide variety of challenging natural language tasks, such as text classification, Question Answering (QA) and Named Entity Recognition and Classification (NERC)  BERT has also been used successfully by other community practitioners for a wide range of NLP-related tasks BIBREF8, BIBREF9 
Regarding the task of data anonymisation in particular, anonymisation systems may follow different approaches and pursue different objectives (Cormode and Srivastava, 2009)  The first objective of these systems is to detect and classify the sensitive information contained in the documents to be anonymised  In order to achieve that, they use rule-based approaches, Machine Learning (ML) approaches, or a combination of both 
Although most of these efforts are for English texts –see, among others, the i2b2 de-identification challenges BIBREF10, BIBREF11, dernon2016deep, or khin2018deep–, other languages are also attracting growing interest  Some examples are mamede2016automated for Portuguese and tveit2004anonymization for Norwegian  With respect to the anonymisation of text written in Spanish, recent studies include medina2018building, hassan2018anonimizacion and garcia2018automating  Most notably, in 2019 the first community challenge about anonymisation of medical documents in Spanish, MEDDOCAN BIBREF3, was held as part of the IberLEF initiative  The winners of the challenge –the Neither-Language-nor-Domain-Experts (NLNDE) BIBREF12– achieved F1-scores as high as 0 975 in the task of sensitive information detection and categorisation by using recurrent neural networks with Conditional Random Field (CRF) output layers 
At the same challenge, mao2019hadoken occupied the 8th position among 18 participants using BERT  According to the description of the system, the authors used BERT-Base Multilingual Cased and an output CRF layer  However, their system is $\sim $3 F1-score points below our implementation without the CRF layer 
Materials and Methods
The aim of this paper is to evaluate BERT's multilingual model and compare it to other established machine-learning algorithms in a specific task: sensitive data detection and classification in Spanish clinical free text  This section describes the data involved in the experiments and the systems evaluated  Finally, we introduce the experimental setup 
Materials and Methods ::: Data
Two datasets are exploited in this article  Both datasets consist of plain text containing clinical narrative written in Spanish, and their respective manual annotations of sensitive information in BRAT BIBREF13 standoff format  In order to feed the data to the different algorithms presented in Section SECREF7, these datasets were transformed to comply with the commonly used BIO sequence representation scheme BIBREF14 
Materials and Methods ::: Data ::: NUBes-PHI
NUBes BIBREF4 is a corpus of around 7,000 real medical reports written in Spanish and annotated with negation and uncertainty information  Before being published, sensitive information had to be manually annotated and replaced for the corpus to be safely shared  In this article, we work with the NUBes version prior to its anonymisation, that is, with the manual annotations of sensitive information  It follows that the version we work with is not publicly available and, due to contractual restrictions, we cannot reveal the provenance of the data  In order to avoid confusion between the two corpus versions, we henceforth refer to the version relevant in this paper as NUBes-PHI (from `NUBes with Personal Health Information') 
NUBes-PHI consists of 32,055 sentences annotated for 11 different sensitive information categories  Overall, it contains 7,818 annotations  The corpus has been randomly split into train (72%), development (8%) and test (20%) sets to conduct the experiments described in this paper  The size of each split and the distribution of the annotations can be consulted in Tables and , respectively 
The majority of sensitive information in NUBes-PHI are temporal expressions (`Date' and `Time'), followed by healthcare facility mentions (`Hospital'), and the age of the patient  Mentions of people are not that frequent, with physician names (`Doctor') occurring much more often than patient names (`Patient')  The least frequent sensitive information types, which account for $\sim $10% of the remaining annotations, consist of the patient's sex, job, and kinship, and locations other than healthcare facilities (`Location')  Finally, the tag `Other' includes, for instance, mentions to institutions unrelated to healthcare and whether the patient is right- or left-handed  It occurs just 36 times

ground_truth: For Article-Entity placement, they consider two baselines: the first one using only salience-based features, and the second baseline checks if the entity appears in the title of the article. 

For Article-Section Placement, they consider two baselines: the first picks the section with the highest lexical similarity to the article, and the second one picks the most frequent section.

B1. The first baseline uses only the salience-based features by Dunietz and Gillick BIBREF11 ., B2. The second baseline assigns the value relevant to a pair INLINEFORM0 , if and only if INLINEFORM1 appears in the title of INLINEFORM2 .

, S1: Pick the section from template INLINEFORM0 with the highest lexical similarity to INLINEFORM1 : S1 INLINEFORM2, S2: Place the news into the most frequent section in INLINEFORM0

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What dataset does this approach achieve state of the art results on?
Context 2:  Changes to adjectives are mainly grammatical, changes to verbs are contextual (e g , changing laufen to rennen, both verbs mean run, but the second refers to running very fast)  For French, 15% of all the changes are substitutions of nouns (for test 2016)  These are again very contextual  For example, the French word travailleur (worker) is replaced by ouvrier (manual worker) in the contexts where tools, machinery or buildings are mentioned  For our analysis we used again spacy 
The information on detected objects is particularly helpful for specific adequacy issues  Figure FIGREF15 demonstrates some such cases  In the first case, the base+att model misses the translation of race car: the German word Rennen translates only the word race  del introduces the word car (Auto) into the translation  Finally, del+obj correctly translates the expression race car (Rennwagen) by exploiting the object information  For French, del translates the source part in a body of water, missing from the base+att translation  del+obj additionally translated the word paddling according to the detected object Paddle 
Source degradation setup
Results of our source degradation experiments are shown in Table TABREF20   A first observation is that – as with the standard setup – the performance of our deliberation models is overall better than that of the base models  The results of the multimodal models differ for German and French  For German, del+obj is the most successful configuration and shows statistically significant improvements over base for all setups  Moreover, for RND and AMB, it shows statistically significant improvements over del  However, especially for RND and AMB, del and del+sum are either the same or slightly worse than base 
For French, all the deliberation models show statistically significant improvements over base (average INLINEFORM0 , INLINEFORM1 ), but the image information added to del only improve scores significantly for test 2018 RND 
This difference in performances for French and German is potentially related to the need of more significant restructurings while translating from English into German  This is where a more complex del+obj architecture is more helpful  This is especially true for RND and AMB setups where blanked words could also be verbs, the part-of-speech most influenced by word order differences between English and German (see the decreasing complexity of translations for del and del+obj for the example (c) in Figure FIGREF21 ) 
To get an insight into the contribution of different contexts to the resolution of blanks, we performed manual analysis of examples coming from the English-German base, del and del+obj setups (50 random examples per setup), where we count correctly translated blanks per system 
The results are shown in Table TABREF27   As expected, they show that the RND and AMB blanks are more difficult to resolve (at most 40% resolved as compared to 61% for PERS)  Translations of the majority of those blanks tend to be guessed by the textual context alone (especially for verbs)  Image information is more helpful for PERS: we observe an increase of 10% in resolved blanks for del+obj as compared to del  However, for PERS the textual context is still enough in the majority of the cases: models tend to associate men with sports or women with cooking and are usually right (see Figure FIGREF21 example (c)) 
The cases where image helps seem to be those with rather generic contexts: see Figure FIGREF21 (b) where enjoying a summer day is not associated with any particular gender and make other models choose homme (man) or femme (woman), and only base+obj chooses enfant (child) (the option closest to the reference) 
In some cases detected objects are inaccurate or not precise enough to be helpful (e g , when an object Person is detected) and can even harm correct translations 
Conclusions
We have proposed a novel approach to multimodal machine translation which makes better use of context, both textual and visual  Our results show that further exploring textual context through deliberation networks already leads to better results than the previous state of the art  Adding visual information, and in particular structural representations of this information, proved beneficial when input text contains noise and the language pair requires substantial restructuring from source to target  Our findings suggest that the combination of a deliberation approach and information from additional modalities is a promising direction for machine translation that is robust to noisy input  Our code and pre-processing scripts are available at https://github com/ImperialNLP/MMT-Delib 
Acknowledgments
The authors thank the anonymous reviewers for their useful feedback  This work was supported by the MultiMT (H2020 ERC Starting Grant No  678017) and MMVC (Newton Fund Institutional Links Grant, ID 352343575) projects  We also thank the annotators for their valuable help 
Context 3: 
Materials and Methods ::: Data ::: The MEDDOCAN corpus
The organisers of the MEDDOCAN shared task BIBREF3 curated a synthetic corpus of clinical cases enriched with sensitive information by health documentalists  In this regard, the MEDDOCAN evaluation scenario could be said to be somewhat far from the real use case the technology developed for the shared task is supposed to be applied in  However, at the moment it also provides the only public means for a rigorous comparison between systems for sensitive health information detection in Spanish texts 
The size of the MEDDOCAN corpus is shown in Table   Compared to NUBes-PHI (Table ), this corpus contains more sensitive information annotations, both in absolute and relative terms 
The sensitive annotation categories considered in MEDDOCAN differ in part from those in NUBes-PHI  Most notably, it contains finer-grained labels for location-related mentions –namely, `Address', `Territory', and `Country'–, and other sensitive information categories that we did not encounter in NUBes-PHI (e g , identifiers, phone numbers, e-mail addresses, etc )  In total, the MEDDOCAN corpus has 21 sensitive information categories  We refer the reader to the organisers' article BIBREF3 for more detailed information about this corpus 
Materials and Methods ::: Systems
Apart from experimenting with a pre-trained BERT model, we have run experiments with other systems and baselines, to compare them and obtain a better perspective about BERT's performance in these datasets 
Materials and Methods ::: Systems ::: Baseline
As the simplest baseline, a sensitive data recogniser and classifier has been developed that consists of regular-expressions and dictionary look-ups  For each category to detect a specific method has been implemented  For instance, the Date, Age, Time and Doctor detectors are based on regular-expressions; Hospital, Sex, Kinship, Location, Patient and Job are looked up in dictionaries  The dictionaries are hand-crafted from the training data available, except for the Patient's case, for which the possible candidates considered are the 100 most common female and male names in Spain according to the Instituto Nacional de Estadística (INE; Spanish Statistical Office) 
Materials and Methods ::: Systems ::: CRF
Conditional Random Fields (CRF) BIBREF15 have been extensively used for tasks of sequential nature  In this paper, we propose as one of the competitive baselines a CRF classifier trained with sklearn-crfsuite for Python 3 5 and the following configuration: algorithm = lbfgs; maximum iterations = 100; c1 = c2 = 0 1; all transitions = true; optimise = false  The features extracted from each token are as follows:
[noitemsep]
prefixes and suffixes of 2 and 3 characters;
the length of the token in characters and the length of the sentence in tokens;
whether the token is all-letters, a number, or a sequence of punctuation marks;
whether the token contains the character `@';
whether the token is the start or end of the sentence;
the token's casing and the ratio of uppercase characters, digits, and punctuation marks to its length;
and, the lemma, part-of-speech tag, and named-entity tag given by ixa-pipes BIBREF16 upon analysing the sentence the token belongs to 
Noticeably, none of the features used to train the CRF classifier is domain-dependent  However, the latter group of features is language dependent 
Materials and Methods ::: Systems ::: spaCy
spaCy is a widely used NLP library that implements state-of-the-art text processing pipelines, including a sequence-labelling pipeline similar to the one described by strubell2017fast  spaCy offers several pre-trained models in Spanish, which perform basic NLP tasks such as Named Entity Recognition (NER)  In this paper, we have trained a new NER model to detect NUBes-PHI labels  For this purpose, the new model uses all the labels of the training corpus coded with its context at sentence level  The network optimisation parameters and dropout values are the ones recommended in the documentation for small datasets  Finally, the model is trained using batches of size 64  No more features are included, so the classifier is language-dependent but not domain-dependent 
Materials and Methods ::: Systems ::: BERT
As introduced earlier, BERT has shown an outstanding performance in NERC-like tasks, improving the start-of-the-art results for almost every dataset and language  We take the same approach here, by using the model BERT-Base Multilingual Cased with a Fully Connected (FC) layer on top to perform a fine-tuning of the whole model for an anonymisation task in Spanish clinical data  Our implementation is built on PyTorch and the PyTorch-Transformers library BIBREF1  The training phase consists in the following steps (roughly depicted in Figure ):
Pre-processing: since we are relying on a pre-trained BERT model, we must match the same configuration by using a specific tokenisation and vocabulary  BERT also needs that the inputs contains special tokens to signal the beginning and the end of each sequence 
Fine-tuning: the pre-processed sequence is fed into the model  BERT outputs the contextual embeddings that encode each of the inputted tokens  This embedding representation for each token is fed into the FC linear layer after a dropout layer (with a 0 1 dropout probability), which in turn outputs the logits for each possible class  The cross-entropy loss function is calculated comparing the logits and the gold labels, and the error is back-propagated to adjust the model parameters 
We have trained the model using an AdamW optimiser BIBREF17 with the learning rate set to 3e-5, as recommended by devlin2018bert, and with a gradient clipping of 1 0  We also applied a learning-rate scheduler that warms up the learning rate from zero to its maximum value as the training progresses, which is also a common practice  For each experiment set proposed below, the training was run with an early-stopping patience of 15 epochs  Then, the model that performed best against the development set was used to produce the reported results 
The experiments were run on a 64-core server with operating system Ubuntu 16 04, 250GB of RAM memory, and 4 GeForce RTX 2080 GPUs with 11GB of memory  The maximum sequence length was set at 500 and the batch size at 12  In this setting, each epoch –a full pass through all the training data– required about 10 minutes to complete 
Materials and Methods ::: Experimental design
We have conducted experiments with BERT in the two datasets of Spanish clinical narrative presented in Section SECREF3 The first experiment set uses NUBes-PHI, a corpus of real medical reports manually annotated with sensitive information  Because this corpus is not publicly available, and in order to compare the BERT-based model to other related published systems, the second set of experiments uses the MEDDOCAN 2019 shared task competition dataset  The following sections provide greater detail about the two experimental setups 
Materials and Methods ::: Experimental design ::: Experiment A: NUBes-PHI
In this experiment set, we evaluate all the systems presented in Section SECREF7, namely, the rule-based baseline, the CRF classifier, the spaCy entity tagger, and BERT  The evaluation comprises three scenarios of increasing difficulty:
[noitemsep]
- Evaluates the performance of the systems at predicting whether each token is sensitive or non-sensitive; that is, the measurements only take into account whether a sensitive token has been recognised or not, regardless of the BIO label and the category assigned  This scenario shows how good a system would be at obfuscating sensitive data (e g , by replacing sensitive tokens with asterisks) 
- We measure the performance of the systems at predicting the sensitive information type of each token –i e , the 11 categories presented in Section SECREF5 or `out'  Detecting entity types correctly is important if a system is going to be used to replace sensitive data by fake data of the same type (e g , random people names) 
- This is the strictest evaluation, as it takes into account both the BIO label and the category assigned to each individual token  Being able to discern between two contiguous sensitive entities of the same type is relevant not only because it is helpful when producing fake replacements, but because it also yields more accurate statistics of the sensitive information present in a given document collection 
The systems are evaluated in terms of micro-average precision, recall and F1-score in all the scenarios 
In addition to the scenarios proposed, a subject worth being studied is the need of labelled data  Manually labelled data is an scarce and expensive resource, which for some application domains or languages is difficult to come by  In order to obtain an estimation of the dependency of each system on the available amount of training data, we have retrained all the compared models using decreasing amounts of data –from 100% of the available training instances to just 1%  The same data subsets have been used to train all the systems  Due to the knowledge transferred from the pre-trained BERT model, the BERT-based model is expected to be more robust to data scarcity than those that start their training from scratch 
Materials and Methods ::: Experimental design ::: Experiment B: MEDDOCAN
In this experiment set, our BERT implementation is compared to several systems that participated in the MEDDOCAN challenge: a CRF classifier BIBREF18, a spaCy entity recogniser BIBREF18, and NLNDE BIBREF12, the winner of the shared task and current state of the art for sensitive information detection and classification in Spanish clinical text  Specifically, we include the results of a domain-independent NLNDE model (S2), and the results of a model enriched with domain-specific embeddings (S3)  Finally, we include the results obtained by mao2019hadoken with a CRF output layer on top of BERT embeddings  MEDDOCAN consists of two scenarios:
[noitemsep]
- This evaluation measures how good a system is at detecting sensitive text spans, regardless of the category assigned to them

ground_truth: the English-German dataset

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the clinical datasets used in the paper?
Context 3: 
- In this scenario, systems are required to match exactly not only the boundaries of each sensitive span, but also the category assigned 
The systems are evaluated in terms of micro-averaged precision, recall and F-1 score  Note that, in contrast to the evaluation in Experiment A, MEDDOCAN measurements are entity-based instead of tokenwise  An exhaustive explanation of the MEDDOCAN evaluation procedure is available online, as well as the official evaluation script, which we used to obtain the reported results 
Results
This section describes the results obtained in the two sets of experiments: NUBes-PHI and MEDDOCAN 
Results ::: Experiment A: NUBes-PHI
Table shows the results of the conducted experiments in NUBes-PHI for all the compared systems  The included baseline serves to give a quick insight about how challenging the data is  With simple regular expressions and gazetteers a precision of 0 853 is obtained  On the other hand, the recall, which directly depends on the coverage provided by the rules and resources, drops to 0 469  Hence, this task is unlikely to be solved without the generalisation capabilities provided by machine-learning and deep-learning models 
Regarding the detection scenario –that is, the scenario concerned with a binary classification to determine whether each individual token conveys sensitive information or not–, it can be observed that BERT outperforms its competitors  A fact worth highlighting is that, according to these results, BERT achieves a precision lower than the rest of the systems (i e , it makes more false positive predictions); in exchange, it obtains a remarkably higher recall  Noticeably, it reaches a recall of 0 979, improving by more than 4 points the second-best system, spaCy 
The table also shows the results for the relaxed metric that only takes into account the entity type detected, regardless of the BIO label (i e , ignoring whether the token is at the beginning or in the middle of a sensitive sequence of tokens)  The conclusions are very similar to those extracted previously, with BERT gaining 2 1 points of F1-score over the CRF based approach  The confusion matrices of the predictions made by CRF, spaCy, and BERT in this scenario are shown in Table   As can bee seen, BERT has less difficulty in predicting correctly less frequent categories, such as `Location', `Job', and `Patient'  One of the most common mistakes according to the confusion matrices is classifying hospital names as `Location' instead of the more accurate `Hospital'; this is hardly a harmful error, given that a hospital is actually a location  Last, the category `Other' is completely leaked by all the compared systems, most likely due to its almost total lack of support in both training and evaluation datasets 
To finish with this experiment set, Table also shows the strict classification precision, recall and F1-score for the compared systems  Despite the fact that, in general, the systems obtain high values, BERT outperforms them again  BERT's F1-score is 1 9 points higher than the next most competitive result in the comparison  More remarkably, the recall obtained by BERT is about 5 points above 
Upon manual inspection of the errors committed by the BERT-based model, we discovered that it has a slight tendency towards producing ill-formed BIO sequences (e g, starting a sensitive span with `Inside' instead of `Begin'; see Table )  We could expect that complementing the BERT-based model with a CRF layer on top would help enforce the emission of valid sequences, alleviating this kind of errors and further improving its results 
Finally, Figure shows the impact of decreasing the amount of training data in the detection scenario  It shows the difference in precision, recall, and F1-score with respect to that obtained using 100% of the training data  A general downward trend can be observed, as one would expect: less training data leads to less accurate predictions  However, the BERT-based model is the most robust to training-data reduction, showing an steadily low performance loss  With 1% of the dataset (230 training instances), the BERT-based model only suffers a striking 7-point F1-score loss, in contrast to the 32 and 39 points lost by the CRF and spaCy models, respectively  This steep performance drop stems to a larger extent from recall decline, which is not that marked in the case of BERT  Overall, these results indicate that the transfer-learning achieved through the BERT multilingual pre-trained model not only helps obtain better results, but also lowers the need of manually labelled data for this application domain 
Results ::: Experiment B: MEDDOCAN
The results of the two MEDDOCAN scenarios –detection and classification– are shown in Table   These results follow the same pattern as in the previous experiments, with the CRF classifier being the most precise of all, and BERT outperforming both the CRF and spaCy classifiers thanks to its greater recall  We also show the results of mao2019hadoken who, despite of having used a BERT-based system, achieve lower scores than our models  The reason why it should be so remain unclear 
With regard to the winner of the MEDDOCAN shared task, the BERT-based model has not improved the scores obtained by neither the domain-dependent (S3) nor the domain-independent (S2) NLNDE model  However, attending to the obtained results, BERT remains only 0 3 F1-score points behind, and would have achieved the second position among all the MEDDOCAN shared task competitors  Taking into account that only 3% of the gold labels remain incorrectly annotated, the task can be considered almost solved, and it is not clear if the differences among the systems are actually significant, or whether they stem from minor variations in initialisation or a long-tail of minor labelling inconsistencies 
Conclusions and Future Work
In this work we have briefly introduced the problems related to data privacy protection in clinical domain  We have also described some of the groundbreaking advances on the Natural Language Processing field due to the appearance of Transformers-based deep-learning architectures and transfer learning from very large general-domain multilingual corpora, focusing our attention in one of its most representative examples, Google's BERT model 
In order to assess the performance of BERT for Spanish clinical data anonymisation, we have conducted several experiments with a BERT-based sequence labelling approach using the pre-trained multilingual BERT model shared by Google as the starting point for the model training  We have compared this BERT-based sequence labelling against other methods and systems  One of the experiments uses the MEDDOCAN 2019 shared task dataset, while the other uses a novel Spanish clinical reports dataset called NUBes-PHI 
The results of the experiments show that, in NUBes-PHI, the BERT-based model outperforms the other systems without requiring any adaptation or domain-specific feature engineering, just by being trained on the provided labelled data  Interestingly, the BERT-based model obtains a remarkably higher recall than the other systems  High recall is a desirable outcome because, when anonymising sensible documents, the accidental leak of sensible data is likely to be more dangerous than the unintended over-obfuscation of non-sensitive text 
Further, we have conducted an additional experiment on this dataset by progressively reducing the training data for all the compared systems  The BERT-based model shows the highest robustness to training-data scarcity, loosing only 7 points of F1-score when trained on 230 instances instead of 21,371  These observation are in line with the results obtained by the NLP community using BERT for other tasks 
The experiments with the MEDDOCAN 2019 shared task dataset follow the same pattern  In this case, the BERT-based model falls 0 3 F1-score points behind the shared task winning system, but it would have achieved the second position in the competition with no further refinement 
Since we have used a pre-trained multilingual BERT model, the same approach is likely to work for other languages just by providing some labelled training data  Further, this is the simplest fine-tuning that can be performed based on BERT  More sophisticated fine-tuning layers could help improve the results  For example, it could be expected that a CRF layer helped enforce better BIO tagging sequence predictions  Precisely, mao2019hadoken participated in the MEDDOCAN competition using a BERT+CRF architecture, but their reported scores are about 3 points lower than our implementation  From the description of their work, it is unclear what the source of this score difference could be 
Further, at the time of writing this paper, new multilingual pre-trained models and Transformer architectures have become available  It would not come as a surprise that these new resources and systems –e g , XLM-RoBERTa BIBREF19 or BETO BIBREF20, a BERT model fully pre-trained on Spanish texts– further advanced the state of the art in this task 
Acknowledgements
This work has been supported by Vicomtech and partially funded by the project DeepReading (RTI2018-096846-B-C21, MCIU/AEI/FEDER,UE) 

ground_truth: MEDDOCAN, NUBes-PHI

MEDDOCAN, NUBes