prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What limitations do the authors demnostrate of their model?
Context 1: Introduction
Neural networks have been successfully used to describe images with text using sequence-to-sequence models BIBREF0  However, the results are simple and dry captions which are one or two phrases long  Humans looking at a painting see more than just objects  Paintings stimulate sentiments, metaphors and stories as well  Therefore, our goal is to have a neural network describe the painting artistically in a style of choice  As a proof of concept, we present a model which generates Shakespearean prose for a given painting as shown in Figure FIGREF1  Accomplishing this task is difficult with traditional sequence to sequence models since there does not exist a large collection of Shakespearean prose which describes paintings: Shakespeare's works describes a single painting shown in Figure FIGREF3  Fortunately we have a dataset of modern English poems which describe images BIBREF1 and line-by-line modern paraphrases of Shakespeare's plays BIBREF2  Our solution is therefore to combine two separately trained models to synthesize Shakespearean prose for a given painting 
Introduction ::: Related work
A general end-to-end approach to sequence learning BIBREF3 makes minimal assumptions on the sequence structure  This model is widely used in tasks such as machine translation, text summarization, conversational modeling, and image captioning  A generative model using a deep recurrent architecture BIBREF0 has also beeen used for generating phrases describing an image  The task of synthesizing multiple lines of poetry for a given image BIBREF1 is accomplished by extracting poetic clues from images  Given the context image, the network associates image attributes with poetic descriptions using a convolutional neural net  The poem is generated using a recurrent neural net which is trained using multi-adversarial training via policy gradient 
Transforming text from modern English to Shakespearean English using text "style transfer" is challenging  An end to end approach using a sequence-to-sequence model over a parallel text corpus BIBREF2 has been proposed based on machine translation  In the absence of a parallel text corpus, generative adversarial networks (GANs) have been used, which simultaneously train two models: a generative model which captures the data distribution, and a discriminative model which evaluates the performance of the generator  Using a target domain language model as a discriminator has also been employed BIBREF4, providing richer and more stable token-level feedback during the learning process  A key challenge in both image and text style transfer is separating content from style BIBREF5, BIBREF6, BIBREF7  Cross-aligned auto-encoder models have focused on style transfer using non-parallel text BIBREF7  Recently, a fine grained model for text style transfer has been proposed BIBREF8 which controls several factors of variation in textual data by using back-translation  This allows control over multiple attributes, such as gender and sentiment, and fine-grained control over the trade-off between content and style 
Methods
We use a total three datasets: two datasets for generating an English poem from an image, and Shakespeare plays and their English translations for text style transfer 
We train a model for generating poems from images based on two datasets BIBREF1  The first dataset consists of image and poem pairs, namely a multi-modal poem dataset (MultiM-Poem), and the second dataset is a large poem corpus, namely a uni-modal poem dataset (UniM-Poem)  The image and poem pairs are extended by adding the nearest three neighbor poems from the poem corpus without redundancy, and an extended image and poem pair dataset is constructed and denoted as MultiM-Poem(Ex)BIBREF1 
We use a collection of line-by-line modern paraphrases for 16 of Shakespeare’s plays BIBREF2, for training a style transfer network from English poems to Shakespearean prose  We use 18,395 sentences from the training data split  We keep 1,218 sentences in the validation data set and 1,462 sentences in our test set 
Methods ::: Image To Poem Actor-Critic Model
For generating a poem from images we use an existing actor-critic architecture BIBREF1  This involves 3 parallel CNNs: an object CNN, sentiment CNN, and scene CNN, for feature extraction  These features are combined with a skip-thought model which provides poetic clues, which are then fed into a sequence-to-sequence model trained by policy gradient with 2 discriminator networks for rewards  This as a whole forms a pipeline that takes in an image and outputs a poem as shown on the top left of Figure FIGREF4  A CNN-RNN generative model acts as an agent  The parameters of this agent define a policy whose execution determines which word is selected as an action  When the agent selects all words in a poem, it receives a reward  Two discriminative networks, shown on the top right of Figure FIGREF4, are defined to serve as rewards concerning whether the generated poem properly describes the input image and whether the generated poem is poetic  The goal of the poem generation model is to generate a sequence of words as a poem for an image to maximize the expected return 
Methods ::: Shakespearizing Poetic Captions
For Shakespearizing modern English texts, we experimented with various types of sequence to sequence models  Since the size of the parallel translation data available is small, we leverage a dictionary providing a mapping between Shakespearean words and modern English words to retrofit pre-trained word embeddings  Incorporating this extra information improves the translation task  The large number of shared word types between the source and target sentences indicates that sharing the representation between them is beneficial 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with Attention
We use a sequence-to-sequence model which consists of a single layer unidrectional LSTM encoder and a single layer LSTM decoder and pre-trained retrofitted word embeddings shared between source and target sentences  We experimented with two different types of attention: global attention BIBREF9, in which the model makes use of the output from the encoder and decoder for the current time step only, and Bahdanau attention BIBREF10, where computing attention requires the output of the decoder from the prior time step  We found that global attention performs better in practice for our task of text style transfer 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with a Pointer Network
Since a pair of corresponding Shakespeare and modern English sentences have significant vocabulary overlap we extend the sequence-to-sequence model mentioned above using pointer networks BIBREF11 that provide location based attention and have been used to enable copying of tokens directly from the input  Moreover, there are lot of proper nouns and rare words which might not be predicted by a vanilla sequence to sequence model 
Methods ::: Shakespearizing Poetic Captions ::: Prediction
For both seq2seq models, we use the attention matrices returned at each decoder time step during inference, to compute the next word in the translated sequence if the decoder output at the current time step is the UNK token  We replace the UNKs in the target output with the highest aligned, maximum attention, source word  The seq2seq model with global attention gives the best results with an average target BLEU score of 29 65 on the style transfer dataset, compared with an average target BLEU score of 26 97 using the seq2seq model with pointer networks 
Results
We perform a qualitative analysis of the Shakespearean prose generated for the input paintings  We conducted a survey, in which we presented famous paintings including those shown in Figures FIGREF1 and FIGREF10 and the corresponding Shakespearean prose generated by the model, and asked 32 students to rate them on the basis of content, creativity and similarity to Shakespearean style on a Likert scale of 1-5  Figure FIGREF12 shows the result of our human evaluation 
The average content score across the paintings is 3 7 which demonstrates that the prose generated is relevant to the painting  The average creativity score is 3 9 which demonstrates that the model captures more than basic objects in the painting successfully using poetic clues in the scene  The average style score is 3 9 which demonstrates that the prose generated is perceived to be in the style of Shakespeare 
We also perform a quantitative analysis of style transfer by generating BLEU scores for the model output using the style transfer dataset  The variation of the BLEU scores with the source sentence lengths is shown in Figure FIGREF11  As expected, the BLEU scores decrease with increase in source sentence lengths 
Results ::: Implementation
All models were trained on Google Colab with a single GPU using Python 3 6 and Tensorflow 2 0  The number of hidden units for the encoder and decoder is 1,576 and 256 for seq2seq with global attention and seq2seq with pointer networks respectively  Adam optimizer was used with the default learning rate of 0 001  The model was trained for 25 epochs  We use pre-trained retrofitted word embeddings of dimension 192 
Results ::: Limitations
Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for "Starry Night" with a low average content score  This happens when the style transfer dataset does not have similar words in the training set of sentences  A solution would be to expand the style transfer dataset, for a better representation of the poem data 
Results ::: Conclusions and Future Work
In conclusion, combining two pipelines with an intermediate representation works well in practice  We observe that a CNN-RNN based image-to-poem net combined with a seq2seq model with parallel text corpus for text style transfer synthesizes Shakespeare-style prose for a given painting  For the seq2seq model used, we observe that it performs better in practice using global attention as compared with local attention  We make our models and code publicly available BIBREF12  In future work we would like to experiment with GANs in the absence of non-parallel datasets, so that we can use varied styles for text style transfer  We would also like to experiment with cross aligned auto-encoders, which form a latent content representation, to efficiently separate style and content 
Context 2: Introduction
Wikipedia is the largest source of open and collaboratively curated knowledge in the world  Introduced in 2001, it has evolved into a reference work with around 5m pages for the English Wikipedia alone  In addition, entities and event pages are updated quickly via collaborative editing and all edits are encouraged to include source citations, creating a knowledge base which aims at being both timely as well as authoritative  As a result, it has become the preferred source of information consumption about entities and events  Moreso, this knowledge is harvested and utilized in building knowledge bases like YAGO BIBREF0 and DBpedia BIBREF1 , and used in applications like text categorization BIBREF2 , entity disambiguation BIBREF3 , entity ranking BIBREF4 and distant supervision BIBREF5 , BIBREF6  
However, not all Wikipedia pages referring to entities (entity pages) are comprehensive: relevant information can either be missing or added with a delay  Consider the city of New Orleans and the state of Odisha which were severely affected by cyclones Hurricane Katrina and Odisha Cyclone, respectively  While Katrina finds extensive mention in the entity page for New Orleans, Odisha Cyclone which has 5 times more human casualties (cf  Figure FIGREF2 ) is not mentioned in the page for Odisha  Arguably Katrina and New Orleans are more popular entities, but Odisha Cyclone was also reported extensively in national and international news outlets  This highlights the lack of important facts in trunk and long-tail entity pages, even in the presence of relevant sources  In addition, previous studies have shown that there is an inherent delay or lag when facts are added to entity pages BIBREF7  
To remedy these problems, it is important to identify information sources that contain novel and salient facts to a given entity page  However, not all information sources are equal  The online presence of major news outlets is an authoritative source due to active editorial control and their articles are also a timely container of facts  In addition, their use is in line with current Wikipedia editing practice, as is shown in BIBREF7 that almost 20% of current citations in all entity pages are news articles  We therefore propose news suggestion as a novel task that enhances entity pages and reduces delay while keeping its pages authoritative 
Existing efforts to populate Wikipedia BIBREF8 start from an entity page and then generate candidate documents about this entity using an external search engine (and then post-process them)  However, such an approach lacks in (a) reproducibility since rankings vary with time with obvious bias to recent news (b) maintainability since document acquisition for each entity has to be periodically performed  To this effect, our news suggestion considers a news article as input, and determines if it is valuable for Wikipedia  Specifically, given an input news article INLINEFORM0 and a state of Wikipedia, the news suggestion problem identifies the entities mentioned in INLINEFORM1 whose entity pages can improve upon suggesting INLINEFORM2   Most of the works on knowledge base acceleration BIBREF9 , BIBREF10 , BIBREF11 , or Wikipedia page generation BIBREF8 rely on high quality input sources which are then utilized to extract textual facts for Wikipedia page population  In this work, we do not suggest snippets or paraphrases but rather entire articles which have a high potential importance for entity pages  These suggested news articles could be consequently used for extraction, summarization or population either manually or automatically – all of which rely on high quality and relevant input sources 
We identify four properties of good news recommendations: salience, relative authority, novelty and placement  First, we need to identify the most salient entities in a news article  This is done to avoid pollution of entity pages with only marginally related news  Second, we need to determine whether the news is important to the entity as only the most relevant news should be added to a precise reference work  To do this, we compute the relative authority of all entities in the news article: we call an entity more authoritative than another if it is more popular or noteworthy in the real world  Entities with very high authority have many news items associated with them and only the most relevant of these should be included in Wikipedia whereas for entities of lower authority the threshold for inclusion of a news article will be lower  Third, a good recommendation should be able to identify novel news by minimizing redundancy coming from multiple news articles  Finally, addition of facts is facilitated if the recommendations are fine-grained, i e , recommendations are made on the section level rather than the page level (placement) 
Approach and Contributions  We propose a two-stage news suggestion approach to entity pages  In the first stage, we determine whether a news article should be suggested for an entity, based on the entity's salience in the news article, its relative authority and the novelty of the article to the entity page  The second stage takes into account the class of the entity for which the news is suggested and constructs section templates from entities of the same class  The generation of such templates has the advantage of suggesting and expanding entity pages that do not have a complete section structure in Wikipedia, explicitly addressing long-tail and trunk entities  Afterwards, based on the constructed template our method determines the best fit for the news article with one of the sections 
We evaluate the proposed approach on a news corpus consisting of 351,982 articles crawled from the news external references in Wikipedia from 73,734 entity pages  Given the Wikipedia snapshot at a given year (in our case [2009-2014]), we suggest news articles that might be cited in the coming years  The existing news references in the entity pages along with their reference date act as our ground-truth to evaluate our approach  In summary, we make the following contributions 
Related Work
As we suggest a new problem there is no current work addressing exactly the same task  However, our task has similarities to Wikipedia page generation and knowledge base acceleration  In addition, we take inspiration from Natural Language Processing (NLP) methods for salience detection 
Wikipedia Page Generation is the problem of populating Wikipedia pages with content coming from external sources  Sauper and Barzilay BIBREF8 propose an approach for automatically generating whole entity pages for specific entity classes  The approach is trained on already-populated entity pages of a given class (e g  `Diseases') by learning templates about the entity page structure (e g  diseases have a treatment section)  For a new entity page, first, they extract documents via Web search using the entity title and the section title as a query, for example `Lung Cancer'+`Treatment'  As already discussed in the introduction, this has problems with reproducibility and maintainability  However, their main focus is on identifying the best paragraphs extracted from the collected documents  They rank the paragraphs via an optimized supervised perceptron model for finding the most representative paragraph that is the least similar to paragraphs in other sections  This paragraph is then included in the newly generated entity page  Taneva and Weikum BIBREF12 propose an approach that constructs short summaries for the long tail  The summaries are called `gems' and the size of a `gem' can be user defined  They focus on generating summaries that are novel and diverse  However, they do not consider any structure of entities, which is present in Wikipedia 
In contrast to BIBREF8 and BIBREF12 , we actually focus on suggesting entire documents to Wikipedia entity pages  These are authoritative documents (news), which are highly relevant for the entity, novel for the entity and in which the entity is salient  Whereas relevance in Sauper and Barzilay is implicitly computed by web page ranking we solve that problem by looking at relative authority and salience of an entity, using the news article and entity page only  As Sauper and Barzilay concentrate on empty entity pages, the problem of novelty of their content is not an issue in their work whereas it is in our case which focuses more on updating entities  Updating entities will be more and more important the bigger an existing reference work is  Both the approaches in BIBREF8 and BIBREF12 (finding paragraphs and summarization) could then be used to process the documents we suggest further  Our concentration on news is also novel 
Knowledge Base Acceleration  In this task, given specific information extraction templates, a given corpus is analyzed in order to find worthwhile mentions of an entity or snippets that match the templates  Balog BIBREF9 , BIBREF10 recommend news citations for an entity  Prior to that, the news articles are classified for their appropriateness for an entity, where as features for the classification task they use entity, document, entity-document and temporal features  The best performing features are those that measure similarity between an entity and the news document  West et al  BIBREF13 consider the problem of knowledge base completion, through question answering and complete missing facts in Freebase based on templates, i e  Frank_Zappa bornIn Baltymore, Maryland 
In contrast, we do not extract facts for pre-defined templates but rather suggest news articles based on their relevance to an entity  In cases of long-tail entities, we can suggest to add a novel section through our abstraction and generation of section templates at entity class level 
Entity Salience  Determining which entities are prominent or salient in a given text has a long history in NLP, sparked by the linguistic theory of Centering BIBREF14   Salience has been used in pronoun and co-reference resolution BIBREF15 , or to predict which entities will be included in an abstract of an article BIBREF11   Frequent features to measure salience include the frequency of an entity in a document, positioning of an entity, grammatical function or internal entity structure (POS tags, head nouns etc )  These approaches are not currently aimed at knowledge base generation or Wikipedia coverage extension but we postulate that an entity's salience in a news article is a prerequisite to the news article being relevant enough to be included in an entity page  We therefore use the salience features in BIBREF11 as part of our model  However, these features are document-internal — we will show that they are not sufficient to predict news inclusion into an entity page and add features of entity authority, news authority and novelty that measure the relations between several entities, between entity and news article as well as between several competing news articles 
Terminology and Problem Definition
We are interested in named entities mentioned in documents  An entity INLINEFORM0 can be identified by a canonical name, and can be mentioned differently in text via different surface forms  We canonicalize these mentions to entity pages in Wikipedia, a method typically known as entity linking  We denote the set of canonicalized entities extracted and linked from a news article INLINEFORM1 as INLINEFORM2   For example, in Figure FIGREF7 , entities are canonicalized into Wikipedia entity pages (e g  Odisha is canonicalized to the corresponding article)  For a collection of news articles INLINEFORM3 , we further denote the resulting set of entities by INLINEFORM4  
Information in an entity page is organized into sections and evolves with time as more content is added  We refer to the state of Wikipedia at a time INLINEFORM0 as INLINEFORM1 and the set of sections for an entity page INLINEFORM2 as its entity profile INLINEFORM3   Unlike news articles, text in Wikipedia could be explicitly linked to entity pages through anchors  The set of entities explicitly referred in text from section INLINEFORM4 is defined as INLINEFORM5   Furthermore, Wikipedia induces a category structure over its entities, which is exploited by knowledge bases like YAGO (e g  Barack_Obama isA Person)  Consequently, each entity page belongs to one or more entity categories or classes INLINEFORM6   Now we can define our news suggestion problem below:
Definition 1 (News Suggestion Problem) Given a set of news articles INLINEFORM0 and set of Wikipedia entity pages INLINEFORM1 (from INLINEFORM2 ) we intend to suggest a news article INLINEFORM3 published at time INLINEFORM4 to entity page INLINEFORM5 and additionally to the most relevant section for the entity page INLINEFORM6  
Approach Overview
We approach the news suggestion problem by decomposing it into two tasks:
AEP: Article–Entity placement
ASP: Article–Section placement
In this first step, for a given entity-news pair INLINEFORM0 , we determine whether the given news article INLINEFORM1 should be suggested (we will refer to this as `relevant') to entity INLINEFORM2   To generate such INLINEFORM3 pairs, we perform the entity linking process, INLINEFORM4 , for INLINEFORM5  
The article–entity placement task (described in detail in Section SECREF16 ) for a pair INLINEFORM0 outputs a binary label (either `non-relevant' or `relevant') and is formalized in Equation EQREF14   DISPLAYFORM0
In the second step, we take into account all `relevant' pairs INLINEFORM0 and find the correct section for article INLINEFORM1 in entity INLINEFORM2 , respectively its profile INLINEFORM3 (see Section SECREF30 )  The article–section placement task, determines the correct section for the triple INLINEFORM4 , and is formalized in Equation EQREF15   DISPLAYFORM0
In the subsequent sections we describe in details how we approach the two tasks for suggesting news articles to entity pages 
News Article Suggestion
In this section, we provide an overview of the news suggestion approach to Wikipedia entity pages (see Figure FIGREF7 )  The approach is split into two tasks: (i) article-entity (AEP) and (ii) article-section (ASP) placement  For a Wikipedia snapshot INLINEFORM0 and a news corpus INLINEFORM1 , we first determine which news articles should be suggested to an entity INLINEFORM2   We will denote our approach for AEP by INLINEFORM3   Finally, we determine the most appropriate section for the ASP task and we denote our approach with INLINEFORM4  
In the following, we describe the process of learning the functions INLINEFORM0 and INLINEFORM1   We introduce features for the learning process, which encode information regarding the entity salience, relative authority and novelty in the case of AEP task  For the ASP task, we measure the overall fit of an article to the entity sections, with the entity being an input from AEP task  Additionally, considering that the entity profiles INLINEFORM2 are incomplete, in the case of a missing section we suggest and expand the entity profiles based on section templates generated from entities of the same class INLINEFORM3 (see Section UID34 ) 
Article–Entity Placement
In this step we learn the function INLINEFORM0 to correctly determine whether INLINEFORM1 should be suggested for INLINEFORM2 , basically a binary classification model (0=`non-relevant' and 1=`relevant')  Note that we are mainly interested in finding the relevant pairs in this task  For every news article, the number of disambiguated entities is around 30 (but INLINEFORM3 is suggested for only two of them on average)  Therefore, the distribution of `non-relevant' and `relevant' pairs is skewed towards the earlier, and by simply choosing the `non-relevant' label we can achieve a high accuracy for INLINEFORM4   Finding the relevant pairs is therefore a considerable challenge 
An article INLINEFORM0 is suggested to INLINEFORM1 by our function INLINEFORM2 if it fulfills the following properties  The entity INLINEFORM3 is salient in INLINEFORM4 (a central concept), therefore ensuring that INLINEFORM5 is about INLINEFORM6 and that INLINEFORM7 is important for INLINEFORM8   Next, given the fact there might be many articles in which INLINEFORM9 is salient, we also look at the reverse property, namely whether INLINEFORM10 is important for INLINEFORM11   We do this by comparing the authority of INLINEFORM12 (which is a measure of popularity of an entity, such as its frequency of mention in a whole corpus) with the authority of its co-occurring entities in INLINEFORM13 , leading to a feature we call relative authority  The intuition is that for an entity that has overall lower authority than its co-occurring entities, a news article is more easily of importance  Finally, if the article we are about to suggest is already covered in the entity profile INLINEFORM14 , we do not wish to suggest redundant information, hence the novelty  Therefore, the learning objective of INLINEFORM15 should fulfill the following properties  Table TABREF21 shows a summary of the computed features for INLINEFORM16  
Salience: entity INLINEFORM0 should be a salient entity in news article INLINEFORM1
Relative Authority: the set of entities INLINEFORM0 with which INLINEFORM1 co-occurs should have higher authority than INLINEFORM2 , making INLINEFORM3 important for INLINEFORM4
Novelty: news article INLINEFORM0 should provide novel information for entity INLINEFORM1 taking into account its profile INLINEFORM2
Baseline Features  As discussed in Section SECREF2 , a variety of features that measure salience of an entity in text are available from the NLP community  We reimplemented the ones in Dunietz and Gillick BIBREF11   This includes a variety of features, e g  positional features, occurrence frequency and the internal POS structure of the entity and the sentence it occurs in  Table 2 in BIBREF11 gives details 
Relative Entity Frequency  Although frequency of mention and positional features play some role in baseline features, their interaction is not modeled by a single feature nor do the positional features encode more than sentence position  We therefore suggest a novel feature called relative entity frequency, INLINEFORM0 , that has three properties : (i) It rewards entities for occurring throughout the text instead of only in some parts of the text, measured by the number of paragraphs it occurs in (ii) it rewards entities that occur more frequently in the opening paragraphs of an article as we model INLINEFORM1 as an exponential decay function  The decay corresponds to the positional index of the news paragraph  This is inspired by the news-specific discourse structure that tends to give short summaries of the most important facts and entities in the opening paragraphs  (iii) it compares entity frequency to the frequency of its co-occurring mentions as the weight of an entity appearing in a specific paragraph, normalized by the sum of the frequencies of other entities in INLINEFORM2   DISPLAYFORM0
where, INLINEFORM0 represents a news paragraph from INLINEFORM1 , and with INLINEFORM2 we indicate the set of all paragraphs in INLINEFORM3   The frequency of INLINEFORM4 in a paragraph INLINEFORM5 is denoted by INLINEFORM6   With INLINEFORM7 and INLINEFORM8 we indicate the number of paragraphs in which entity INLINEFORM9 occurs, and the total number of paragraphs, respectively 
Relative Authority  In this case, we consider the comparative relevance of the news article to the different entities occurring in it  As an example, let us consider the meeting of the Sudanese bishop Elias Taban with Hillary Clinton  Both entities are salient for the meeting  However, in Taban's Wikipedia page, this meeting is discussed prominently with a corresponding news reference, whereas in Hillary Clinton's Wikipedia page it is not reported at all  We believe this is not just an omission in Clinton's page but mirrors the fact that for the lesser known Taban the meeting is big news whereas for the more famous Clinton these kind of meetings are a regular occurrence, not all of which can be reported in what is supposed to be a selection of the most important events for her  Therefore, if two entities co-occur, the news is more relevant for the entity with the lower a priori authority 
The a priori authority of an entity (denoted by INLINEFORM0 ) can be measured in several ways  We opt for two approaches: (i) probability of entity INLINEFORM1 occurring in the corpus INLINEFORM2 , and (ii) authority assessed through centrality measures like PageRank BIBREF16   For the second case we construct the graph INLINEFORM3 consisting of entities in INLINEFORM4 and news articles in INLINEFORM5 as vertices  The edges are established between INLINEFORM6 and entities in INLINEFORM7 , that is INLINEFORM8 , and the out-links from INLINEFORM9 , that is INLINEFORM10 (arrows present the edge direction) 
Starting from a priori authority, we proceed to relative authority by comparing the a priori authority of co-occurring entities in INLINEFORM0   We define the relative authority of INLINEFORM1 as the proportion of co-occurring entities INLINEFORM2 that have a higher a priori authority than INLINEFORM3 (see Equation EQREF28   DISPLAYFORM0
As we might run the danger of not suggesting any news articles for entities with very high a priori authority (such as Clinton) due to the strict inequality constraint, we can relax the constraint such that the authority of co-occurring entities is above a certain threshold 
News Domain Authority  The news domain authority addresses two main aspects  Firstly, if bundled together with the relative authority feature, we can ensure that dependent on the entity authority, we suggest news from authoritative sources, hence ensuring the quality of suggested articles  The second aspect is in a news streaming scenario where multiple news domains report the same event — ideally only articles coming from authoritative sources would fulfill the conditions for the news suggestion task 
The news domain authority is computed based on the number of news references in Wikipedia coming from a particular news domain INLINEFORM0   This represents a simple prior that a news article INLINEFORM1 is from domain INLINEFORM2 in corpus INLINEFORM3   We extract the domains by taking the base URLs from the news article URLs 
An important feature when suggesting an article INLINEFORM0 to an entity INLINEFORM1 is the novelty of INLINEFORM2 w r t the already existing entity profile INLINEFORM3   Studies BIBREF17 have shown that on comparable collections to ours (TREC GOV2) the number of duplicates can go up to INLINEFORM4   This figure is likely higher for major events concerning highly authoritative entities on which all news media will report 
Given an entity INLINEFORM0 and the already added news references INLINEFORM1 up to year INLINEFORM2 , the novelty of INLINEFORM3 at year INLINEFORM4 is measured by the KL divergence between the language model of INLINEFORM5 and articles in INLINEFORM6   We combine this measure with the entity overlap of INLINEFORM7 and INLINEFORM8   The novelty value of INLINEFORM9 is given by the minimal divergence value  Low scores indicate low novelty for the entity profile INLINEFORM10  
N(n|e) = n'Nt-1{DKL((n') || (n)) + DKL((N) || (n)) 
DKL((n') || (n))  (1-) jaccard((n'),(n))} where INLINEFORM0 is the KL divergence of the language models ( INLINEFORM1 and INLINEFORM2 ), whereas INLINEFORM3 is the mixing weight ( INLINEFORM4 ) between the language models INLINEFORM5 and the entity overlap in INLINEFORM6 and INLINEFORM7  
Here we introduce the evaluation setup and analyze the results for the article–entity (AEP) placement task  We only report the evaluation metrics for the `relevant' news-entity pairs  A detailed explanation on why we focus on the `relevant' pairs is provided in Section SECREF16  
Baselines  We consider the following baselines for this task 
B1  The first baseline uses only the salience-based features by Dunietz and Gillick BIBREF11  
B2  The second baseline assigns the value relevant to a pair INLINEFORM0 , if and only if INLINEFORM1 appears in the title of INLINEFORM2  
Learning Models  We use Random Forests (RF) BIBREF23   We learn the RF on all computed features in Table TABREF21   The optimization on RF is done by splitting the feature space into multiple trees that are considered as ensemble classifiers  Consequently, for each classifier it computes the margin function as a measure of the average count of predicting the correct class in contrast to any other class  The higher the margin score the more robust the model 
Metrics  We compute precision P, recall R and F1 score for the relevant class  For example, precision is the number of news-entity pairs we correctly labeled as relevant compared to our ground truth divided by the number of all news-entity pairs we labeled as relevant 
The following results measure the effectiveness of our approach in three main aspects: (i) overall performance of INLINEFORM0 and comparison to baselines, (ii) robustness across the years, and (iii) optimal model for the AEP placement task 
Performance  Figure FIGREF55 shows the results for the years 2009 and 2013, where we optimized the learning objective with instances from year INLINEFORM0 and evaluate on the years INLINEFORM1 (see Section SECREF46 )  The results show the precision–recall curve  The red curve shows baseline B1 BIBREF11 , and the blue one shows the performance of INLINEFORM2   The curve shows for varying confidence scores (high to low) the precision on labeling the pair INLINEFORM3 as `relevant'  In addition, at each confidence score we can compute the corresponding recall for the `relevant' label  For high confidence scores on labeling the news-entity pairs, the baseline B1 achieves on average a precision score of P=0 50, while INLINEFORM4 has P=0 93  We note that with the drop in the confidence score the corresponding precision and recall values drop too, and the overall F1 score for B1 is around F1=0 2, in contrast we achieve an average score of F1=0 67 
It is evident from Figure FIGREF55 that for the years 2009 and 2013, INLINEFORM0 significantly outperforms the baseline B1  We measure the significance through the t-test statistic and get a p-value of INLINEFORM1   The improvement we achieve over B1 in absolute numbers, INLINEFORM2 P=+0 5 in terms of precision for the years between 2009 and 2014, and a similar improvement in terms of F1 score  The improvement for recall is INLINEFORM3 R=+0 4  The relative improvement over B1 for P and F1 is almost 1 8 times better, while for recall we are 3 5 times better  In Table TABREF58 we show the overall scores for the evaluation metrics for B1 and INLINEFORM4 
Context 3: Introduction
Deep learning systems have shown a lot of promise for extractive Question Answering (QA), with performance comparable to humans when large scale data is available  However, practitioners looking to build QA systems for specific applications may not have the resources to collect tens of thousands of questions on corpora of their choice  At the same time, state-of-the-art machine reading systems do not lend well to low-resource QA settings where the number of labeled question-answer pairs are limited (c f  Table 2 )  Semi-supervised QA methods like BIBREF0 aim to improve this performance by leveraging unlabeled data which is easier to collect 
In this work, we present a semi-supervised QA system which requires the end user to specify a set of base documents and only a small set of question-answer pairs over a subset of these documents  Our proposed system consists of three stages  First, we construct cloze-style questions (predicting missing spans of text) from the unlabeled corpus; next, we use the generated clozes to pre-train a powerful neural network model for extractive QA BIBREF1 , BIBREF2 ; and finally, we fine-tune the model on the small set of provided QA pairs 
Our cloze construction process builds on a typical writing phenomenon and document structure: an introduction precedes and summarizes the main body of the article  Many large corpora follow such a structure, including Wikipedia, academic papers, and news articles  We hypothesize that we can benefit from the un-annotated corpora to better answer various questions – at least ones that are lexically similar to the content in base documents and directly require factual information 
We apply the proposed system on three datasets from different domains – SQuAD BIBREF3 , TriviaQA-Web BIBREF4 and the BioASQ challenge BIBREF5   We observe significant improvements in a low-resource setting across all three datasets  For SQuAD and TriviaQA, we attain an F1 score of more than 50% by merely using 1% of the training data  Our system outperforms the approaches for semi-supervised QA presented in BIBREF0 , and a baseline which uses the same unlabeled data but with a language modeling objective for pretraining  In the BioASQ challenge, we outperform the best performing system from previous year's challenge, improving over a baseline which does transfer learning from the SQuAD dataset  Our analysis reveals that questions which ask for factual information and match to specific parts of the context documents benefit the most from pretraining on automatically constructed clozes 
Related Work
Semi-supervised learning augments the labeled dataset $L$ with a potentially larger unlabeled dataset $U$   BIBREF0 presented a model, GDAN, which trained an auxiliary neural network to generate questions from passages by reinforcement learning, and augment the labeled dataset with the generated questions to train the QA model  Here we use a much simpler heuristic to generate the auxiliary questions, which also turns out to be more effective as we show superior performance compared to GDAN  Several approaches have been suggested for generating natural questions BIBREF6 , BIBREF7 , BIBREF8 , however none of them show a significant improvement of using the generated questions in a semi-supervised setting  Recent papers also use unlabeled data for QA by training large language models and extracting contextual word vectors from them to input to the QA model BIBREF9 , BIBREF10 , BIBREF11   The applicability of this method in the low-resource setting is unclear as the extra inputs increase the number of parameters in the QA model, however, our pretraining can be easily applied to these models as well 
Domain adaptation (and Transfer learning) leverage existing large scale datasets from a source domain (or task) to improve performance on a target domain (or task)  For deep learning and QA, a common approach is to pretrain on the source dataset and then fine-tune on the target dataset BIBREF12 , BIBREF13   BIBREF14 used SQuAD as a source for the target BioASQ dataset, and BIBREF15 used Book Test BIBREF16 as source for the target SQuAD dataset  BIBREF17 transfer learned model layers from the tasks of sequence labeling, text classification and relation classification to show small improvements on SQuAD  All these works use manually curated source datatset, which in themselves are expensive to collect  Instead, we show that it is possible to automatically construct the source dataset from the same domain as the target, which turns out to be more beneficial in terms of performance as well (c f  Section "Experiments & Results" )  Several cloze datasets have been proposed in the literature which use heuristics for construction BIBREF18 , BIBREF19 , BIBREF20   We further see the usability of such a dataset in a semi-supervised setting 
Methodology
Our system comprises of following three steps:
Cloze generation: Most of the documents typically follow a template, they begin with an introduction that provides an overview and a brief summary for what is to follow  We assume such a structure while constructing our cloze style questions  When there is no clear demarcation, we treat the first $K\%$ (hyperparameter, in our case 20%) of the document as the introduction  While noisy, this heuristic generates a large number of clozes given any corpus, which we found to be beneficial for semi-supervised learning despite the noise 
We use a standard NLP pipeline based on Stanford CoreNLP (for SQuAD, TrivaQA and PubMed) and the BANNER Named Entity Recognizer (only for PubMed articles) to identify entities and phrases  Assume that a document comprises of introduction sentences $\lbrace q_1, q_2,     q_n\rbrace $ , and the remaining passages $\lbrace p_1, p_2,    p_m\rbrace $   Additionally, let's say that each sentence $q_i$ in introduction is composed of words $\lbrace w_1, w_2,     w_{l_{q_i}}\rbrace $ , where $l_{q_i}$ is the length of $q_i$   We consider a $\text{match} (q_i, p_j)$ , if there is an exact string match of a sequence of words $\lbrace w_k, w_{k+1},    w_{l_{q_i}}\rbrace $ between the sentence $q_i$ and passage $p_j$   If this sequence is either a noun phrase, verb phrase, adjective phrase or a named entity in $\lbrace p_1, p_2,    p_m\rbrace $0 , as recognized by CoreNLP or BANNER, we select it as an answer span $\lbrace p_1, p_2,    p_m\rbrace $1   Additionally, we use $\lbrace p_1, p_2,    p_m\rbrace $2 as the passage $\lbrace p_1, p_2,    p_m\rbrace $3 and form a cloze question $\lbrace p_1, p_2,    p_m\rbrace $4 from the answer bearing sentence $\lbrace p_1, p_2,    p_m\rbrace $5 by replacing $\lbrace p_1, p_2,    p_m\rbrace $6 with a placeholder  As a result, we obtain passage-question-answer ( $\lbrace p_1, p_2,    p_m\rbrace $7 ) triples (Table 1 shows an example)  As a post-processing step, we prune out $\lbrace p_1, p_2, 

ground_truth: Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer

we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for "Starry Night" with a low average content score

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What baseline model is used?
Context 2:  Finally, for B2 we achieve much poorer performance, with average scores of P=0 21, R=0 20 and F1=0 21 
Robustness  In Table TABREF58 , we show the overall performance for the years between 2009 and 2013  An interesting observation we make is that we have a very robust performance and the results are stable across the years  If we consider the experimental setup, where for year INLINEFORM0 we optimize the learning objective with only 74k training instances and evaluate on the rest of the instances, it achieves a very good performance  We predict with F1=0 68 the remaining 469k instances for the years INLINEFORM1  
The results are particularly promising considering the fact that the distribution between our two classes is highly skewed  On average the number of `relevant' pairs account for only around INLINEFORM0 of all pairs  A good indicator to support such a statement is the kappa (denoted by INLINEFORM1 ) statistic  INLINEFORM2 measures agreement between the algorithm and the gold standard on both labels while correcting for chance agreement (often expected due to extreme distributions)  The INLINEFORM3 scores for B1 across the years is on average INLINEFORM4 , while for INLINEFORM5 we achieve a score of INLINEFORM6 (the maximum score for INLINEFORM7 is 1) 
In Figure FIGREF60 we show the impact of the individual feature groups that contribute to the superior performance in comparison to the baselines  Relative entity frequency from the salience feature, models the entity salience as an exponentially decaying function based on the positional index of the paragraph where the entity appears  The performance of INLINEFORM0 with relative entity frequency from the salience feature group is close to that of all the features combined  The authority and novelty features account to a further improvement in terms of precision, by adding roughly a 7%-10% increase  However, if both feature groups are considered separately, they significantly outperform the baseline B1 
Article–Section Placement
We model the ASP placement task as a successor of the AEP task  For all the `relevant' news entity pairs, the task is to determine the correct entity section  Each section in a Wikipedia entity page represents a different topic  For example, Barack Obama has the sections `Early Life', `Presidency', `Family and Personal Life' etc  However, many entity pages have an incomplete section structure  Incomplete or missing sections are due to two Wikipedia properties  First, long-tail entities miss information and sections due to their lack of popularity  Second, for all entities whether popular or not, certain sections might occur for the first time due to real world developments  As an example, the entity Germanwings did not have an `Accidents' section before this year's disaster, which was the first in the history of the airline 
Even if sections are missing for certain entities, similar sections usually occur in other entities of the same class (e g  other airlines had disasters and therefore their pages have an accidents section)  We exploit such homogeneity of section structure and construct templates that we use to expand entity profiles  The learning objective for INLINEFORM0 takes into account the following properties:
Section-templates: account for incomplete section structure for an entity profile INLINEFORM0 by constructing section templates INLINEFORM1 from an entity class INLINEFORM2
Overall fit: measures the overall fit of a news article to sections in the section templates INLINEFORM0
Given the fact that entity profiles are often incomplete, we construct section templates for every entity class  We group entities based on their class INLINEFORM0 and construct section templates INLINEFORM1   For different entity classes, e g  Person and Location, the section structure and the information represented in those section varies heavily  Therefore, the section templates are with respect to the individual classes in our experimental setup (see Figure FIGREF42 )  DISPLAYFORM0
Generating section templates has two main advantages  Firstly, by considering class-based profiles, we can overcome the problem of incomplete individual entity profiles and thereby are able to suggest news articles to sections that do not yet exist in a specific entity INLINEFORM0   The second advantage is that we are able to canonicalize the sections, i e  `Early Life' and `Early Life and Childhood' would be treated similarly 
To generate the section template INLINEFORM0 , we extract all sections from entities of a given type INLINEFORM1 at year INLINEFORM2   Next, we cluster the entity sections, based on an extended version of k–means clustering BIBREF18 , namely x–means clustering introduced in Pelleg et al  which estimates the number of clusters efficiently BIBREF19   As a similarity metric we use the cosine similarity computed based on the tf–idf models of the sections  Using the x–means algorithm we overcome the requirement to provide the number of clusters k beforehand  x–means extends the k–means algorithm, such that a user only specifies a range [ INLINEFORM3 , INLINEFORM4 ] that the number of clusters may reasonably lie in 
The learning objective of INLINEFORM0 is to determine the overall fit of a news article INLINEFORM1 to one of the sections in a given section template INLINEFORM2   The template is pre-determined by the class of the entity for which the news is suggested as relevant by INLINEFORM3   In all cases, we measure how well INLINEFORM4 fits each of the sections INLINEFORM5 as well as the specific entity section INLINEFORM6   The section profiles in INLINEFORM7 represent the aggregated entity profiles from all entities of class INLINEFORM8 at year INLINEFORM9  
To learn INLINEFORM0 we rely on a variety of features that consider several similarity aspects as shown in Table TABREF31   For the sake of simplicity we do not make the distinction in Table TABREF31 between the individual entity section and class-based section similarities, INLINEFORM1 and INLINEFORM2 , respectively  Bear in mind that an entity section INLINEFORM3 might be present at year INLINEFORM4 but not at year INLINEFORM5 (see for more details the discussion on entity profile expansion in Section UID69 ) 
Topic  We use topic similarities to ensure (i) that the content of INLINEFORM0 fits topic-wise with a specific section text and (ii) that it has a similar topic to previously referred news articles in that section  In a pre-processing stage we compute the topic models for the news articles, entity sections INLINEFORM1 and the aggregated class-based sections in INLINEFORM2   The topic models are computed using LDA BIBREF20   We only computed a single topic per article/section as we are only interested in topic term overlaps between article and sections  We distinguish two main features: the first feature measures the overlap of topic terms between INLINEFORM3 and the entity section INLINEFORM4 and INLINEFORM5 , and the second feature measures the overlap of the topic model of INLINEFORM6 against referred news articles in INLINEFORM7 at time INLINEFORM8  
Syntactic  These features represent a mechanism for conveying the importance of a specific text snippet, solely based on the frequency of specific POS tags (i e  NNP, CD etc ), as commonly used in text summarization tasks  Following the same intuition as in BIBREF8 , we weigh the importance of articles by the count of specific POS tags  We expect that for different sections, the importance of POS tags will vary  We measure the similarity of POS tags in a news article against the section text  Additionally, we consider bi-gram and tri-gram POS tag overlap  This exploits similarity in syntactical patterns between the news and section text 
Lexical  As lexical features, we measure the similarity of INLINEFORM0 against the entity section text INLINEFORM1 and the aggregate section text INLINEFORM2   Further, we distinguish between the overall similarity of INLINEFORM3 and that of the different news paragraphs ( INLINEFORM4 which denotes the paragraphs of INLINEFORM5 up to the 5th paragraph)  A higher similarity on the first paragraphs represents a more confident indicator that INLINEFORM6 should be suggested to a specific section INLINEFORM7   We measure the similarity based on two metrics: (i) the KL-divergence between the computed language models and (ii) cosine similarity of the corresponding paragraph text INLINEFORM8 and section text 
Entity-based  Another feature set we consider is the overlap of named entities and their corresponding entity classes  For different entity sections, we expect to find a particular set of entity classes that will correlate with the section, e g  `Early Life' contains mostly entities related to family, school, universities etc 
Frequency  Finally, we gather statistics about the number of entities, paragraphs, news article length, top– INLINEFORM0 entities and entity classes, and the frequency of different POS tags  Here we try to capture patterns of articles that are usually cited in specific sections 
Evaluation Plan
In this section we outline the evaluation plan to verify the effectiveness of our learning approaches  To evaluate the news suggestion problem we are faced with two challenges 
What comprises the ground truth for such a task ?
How do we construct training and test splits given that entity pages consists of text added at different points in time ?
Consider the ground truth challenge  Evaluating if an arbitrary news article should be included in Wikipedia is both subjective and difficult for a human if she is not an expert  An invasive approach, which was proposed by Barzilay and Sauper BIBREF8 , adds content directly to Wikipedia and expects the editors or other users to redact irrelevant content over a period of time  The limitations of such an evaluation technique is that content added to long-tail entities might not be evaluated by informed users or editors in the experiment time frame  It is hard to estimate how much time the added content should be left on the entity page  A more non-invasive approach could involve crowdsourcing of entity and news article pairs in an IR style relevance assessment setup  The problem of such an approach is again finding knowledgeable users or experts for long-tail entities  Thus the notion of relevance of a news recommendation is challenging to evaluate in a crowd setup 
We take a slightly different approach by making an assumption that the news articles already present in Wikipedia entity pages are relevant  To this extent, we extract a dataset comprising of all news articles referenced in entity pages (details in Section SECREF40 )  At the expense of not evaluating the space comprising of news articles absent in Wikipedia, we succeed in (i) avoiding restrictive assumptions about the quality of human judgments, (ii) being invasive and polluting Wikipedia, and (iii) deriving a reusable test bed for quicker experimentation 
The second challenge of construction of training and test set separation is slightly easier and is addressed in Section SECREF46  
Datasets
The datasets we use for our experimental evaluation are directly extracted from the Wikipedia entity pages and their revision history  The generated data represents one of the contributions of our paper  The datasets are the following:
Entity Classes  We focus on a manually predetermined set of entity classes for which we expect to have news coverage  The number of analyzed entity classes is 27, including INLINEFORM0 entities with at least one news reference  The entity classes were selected from the DBpedia class ontology  Figure FIGREF42 shows the number of entities per class for the years (2009-2014) 
News Articles  We extract all news references from the collected Wikipedia entity pages  The extracted news references are associated with the sections in which they appear  In total there were INLINEFORM0 news references, and after crawling we end up with INLINEFORM1 successfully crawled news articles  The details of the news article distribution, and the number of entities and sections from which they are referred are shown in Table TABREF44  
Article-Entity Ground-truth  The dataset comprises of the news and entity pairs INLINEFORM0   News-entity pairs are relevant if the news article is referenced in the entity page  Non-relevant pairs (i e  negative training examples) consist of news articles that contain an entity but are not referenced in that entity's page  If a news article INLINEFORM1 is referred from INLINEFORM2 at year INLINEFORM3 , the features are computed taking into account the entity profiles at year INLINEFORM4  
Article-Section Ground-truth  The dataset consists of the triple INLINEFORM0 , where INLINEFORM1 , where we assume that INLINEFORM2 has already been determined as relevant  We therefore have a multi-class classification problem where we need to determine the section of INLINEFORM3 where INLINEFORM4 is cited  Similar to the article-entity ground truth, here too the features compute the similarity between INLINEFORM5 , INLINEFORM6 and INLINEFORM7  
Data Pre-Processing
We POS-tag the news articles and entity profiles INLINEFORM0 with the Stanford tagger BIBREF21   For entity linking the news articles, we use TagMe! BIBREF22 with a confidence score of 0 3  On a manual inspection of a random sample of 1000 disambiguated entities, the accuracy is above 0 9  On average, the number of entities per news article is approximately 30  For entity linking the entity profiles, we simply follow the anchor text that refers to Wikipedia entities 
Train and Testing Evaluation Setup
We evaluate the generated supervised models for the two tasks, AEP and ASP, by splitting the train and testing instances  It is important to note that for the pairs INLINEFORM0 and the triple INLINEFORM1 , the news article INLINEFORM2 is referenced at time INLINEFORM3 by entity INLINEFORM4 , while the features take into account the entity profile at time INLINEFORM5   This avoids any `overlapping' content between the news article and the entity page, which could affect the learning task of the functions INLINEFORM6 and INLINEFORM7   Table TABREF47 shows the statistics of train and test instances  We learn the functions at year INLINEFORM8 and test on instances for the years greater than INLINEFORM9   Please note that we do not show the performance for year 2014 as we do not have data for 2015 for evaluation 
Article-Section Placement
Here we show the evaluation setup for ASP task and discuss the results with a focus on three main aspects, (i) the overall performance across the years, (ii) the entity class specific performance, and (iii) the impact on entity profile expansion by suggesting missing sections to entities based on the pre-computed templates 
Baselines  To the best of our knowledge, we are not aware of any comparable approach for this task  Therefore, the baselines we consider are the following:
S1: Pick the section from template INLINEFORM0 with the highest lexical similarity to INLINEFORM1 : S1 INLINEFORM2
S2: Place the news into the most frequent section in INLINEFORM0
Learning Models  We use Random Forests (RF) BIBREF23 and Support Vector Machines (SVM) BIBREF24   The models are optimized taking into account the features in Table TABREF31   In contrast to the AEP task, here the scale of the number of instances allows us to learn the SVM models  The SVM model is optimized using the INLINEFORM0 loss function and uses the Gaussian kernels 
Metrics  We compute precision P as the ratio of news for which we pick a section INLINEFORM0 from INLINEFORM1 and INLINEFORM2 conforms to the one in our ground-truth (see Section SECREF40 )  The definition of recall R and F1 score follows from that of precision 
Figure FIGREF66 shows the overall performance and a comparison of our approach (when INLINEFORM0 is optimized using SVM) against the best performing baseline S2  With the increase in the number of training instances for the ASP task the performance is a monotonically non-decreasing function  For the year 2009, we optimize the learning objective of INLINEFORM1 with around 8% of the total instances, and evaluate on the rest  The performance on average is around P=0 66 across all classes  Even though for many classes the performance is already stable (as we will see in the next section), for some classes we improve further  If we take into account the years between 2010 and 2012, we have an increase of INLINEFORM2 P=0 17, with around 70% of instances used for training and the remainder for evaluation  For the remaining years the total improvement is INLINEFORM3 P=0 18 in contrast to the performance at year 2009 
On the other hand, the baseline S1 has an average precision of P=0 12  The performance across the years varies slightly, with the year 2011 having the highest average precision of P=0 13  Always picking the most frequent section as in S2, as shown in Figure FIGREF66 , results in an average precision of P=0 17, with a uniform distribution across the years 
Here we show the performance of INLINEFORM0 decomposed for the different entity classes  Specifically we analyze the 27 classes in Figure FIGREF42   In Table TABREF68 , we show the results for a range of years (we omit showing all years due to space constraints)  For illustration purposes only, we group them into four main classes ( INLINEFORM1 Person, Organization, Location, Event INLINEFORM2 ) and into the specific sub-classes shown in the second column in Table TABREF68   For instance, the entity classes OfficeHolder and Politician are aggregated into Person–Politics 
It is evident that in the first year the performance is lower in contrast to the later years  This is due to the fact that as we proceed, we can better generalize and accurately determine the correct fit of an article INLINEFORM0 into one of the sections from the pre-computed templates INLINEFORM1   The results are already stable for the year range INLINEFORM2   For a few Person sub-classes, e g  Politics, Entertainment, we achieve an F1 score above 0 9  These additionally represent classes with a sufficient number of training instances for the years INLINEFORM3   The lowest F1 score is for the Criminal and Television classes  However, this is directly correlated with the insufficient number of instances 
The baseline approaches for the ASP task perform poorly  S1, based on lexical similarity, has a varying performance for different entity classes  The best performance is achieved for the class Person – Politics, with P=0 43  This highlights the importance of our feature choice and that the ASP cannot be considered as a linear function, where the maximum similarity yields the best results  For different entity classes different features and combination of features is necessary  Considering that S2 is the overall best performing baseline, through our approach INLINEFORM0 we have a significant improvement of over INLINEFORM1 P=+0 64 
The models we learn are very robust and obtain high accuracy, fulfilling our pre-condition for accurate news suggestions into the entity sections  We measure the robustness of INLINEFORM0 through the INLINEFORM1 statistic  In this case, we have a model with roughly 10 labels (corresponding to the number of sections in a template INLINEFORM2 )  The score we achieve shows that our model predicts with high confidence with INLINEFORM3  
The last analysis is the impact we have on expanding entity profiles INLINEFORM0 with new sections  Figure FIGREF70 shows the ratio of sections for which we correctly suggest an article INLINEFORM1 to the right section in the section template INLINEFORM2   The ratio here corresponds to sections that are not present in the entity profile at year INLINEFORM3 , that is INLINEFORM4   However, given the generated templates INLINEFORM5 , we can expand the entity profile INLINEFORM6 with a new section at time INLINEFORM7   In details, in the absence of a section at time INLINEFORM8 , our model trains well on similar sections from the section template INLINEFORM9 , hence we can predict accurately the section and in this case suggest its addition to the entity profile  With time, it is obvious that the expansion rate decreases at later years as the entity profiles become more `complete' 
This is particularly interesting for expanding the entity profiles of long-tail entities as well as updating entities with real-world emerging events that are added constantly  In many cases such missing sections are present at one of the entities of the respective entity class INLINEFORM0   An obvious case is the example taken in Section SECREF16 , where the `Accidents' is rather common for entities of type Airline  However, it is non-existent for some specific entity instances, i e Germanwings airline 
Through our ASP approach INLINEFORM0 , we are able to expand both long-tail and trunk entities  We distinguish between the two types of entities by simply measuring their section text length  The real distribution in the ground truth (see Section SECREF40 ) is 27% and 73% are long-tail and trunk entities, respectively  We are able to expand the entity profiles for both cases and all entity classes without a significant difference, with the only exception being the class Creative Work, where we expand significantly more trunk entities 
Conclusion and Future Work
In this work, we have proposed an automated approach for the novel task of suggesting news articles to Wikipedia entity pages to facilitate Wikipedia updating  The process consists of two stages  In the first stage, article–entity placement, we suggest news articles to entity pages by considering three main factors, such as entity salience in a news article, relative authority and novelty of news articles for an entity page  In the second stage, article–section placement, we determine the best fitting section in an entity page  Here, we remedy the problem of incomplete entity section profiles by constructing section templates for specific entity classes  This allows us to add missing sections to entity pages  We carry out an extensive experimental evaluation on 351,983 news articles and 73,734 entities coming from 27 distinct entity classes  For the first stage, we achieve an overall performance with P=0 93, R=0 514 and F1=0 676, outperforming our baseline competitors significantly  For the second stage, we show that we can learn incrementally to determine the correct section for a news article based on section templates  The overall performance across different classes is P=0 844, R=0 885 and F1=0 860 
In the future, we will enhance our work by extracting facts from the suggested news articles  Results suggest that the news content cited in entity pages comes from the first paragraphs  However, challenging task such as the canonicalization and chronological ordering of facts, still remain 
Context 3:   p_m\rbrace $8 triples where the word overlap between the question (Q) and passage (P) is less than 2 words (after excluding the stop words) 
The process relies on the fact that answer candidates from the introduction are likely to be discussed in detail in the remainder of the article  In effect, the cloze question from the introduction and the matching paragraph in the body forms a question and context passage pair  We create two cloze datasets, one each from Wikipedia corpus (for SQuAD and TriviaQA) and PUBMed academic papers (for the BioASQ challenge), consisting of 2 2M and 1M clozes respectively  From analyzing the cloze data manually, we were able to answer 76% times for the Wikipedia set and 80% times for the PUBMed set using the information in the passage  In most cases the cloze paraphrased the information in the passage, which we hypothesized to be a useful signal for the downstream QA task 
We also investigate the utility of forming subsets of the large cloze corpus, where we select the top passage-question-answer triples, based on the different criteria, like i) jaccard similarity of answer bearing sentence in introduction and the passage ii) the tf-idf scores of answer candidates and iii) the length of answer candidates  However, we empirically find that we were better off using the entire set rather than these subsets 
Pre-training: We make use of the generated cloze dataset to pre-train an expressive neural network designed for the task of reading comprehension  We work with two publicly available neural network models – the GA Reader BIBREF2 (to enable comparison with prior work) and BiDAF + Self-Attention (SA) model from BIBREF1 (which is among the best performing models on SQuAD and TriviaQA)  After pretraining, the performance of BiDAF+SA on a dev set of the (Wikipedia) cloze questions is 0 58 F1 score and 0 55 Exact Match (EM) score  This implies that the cloze corpus is neither too easy, nor too difficult to answer 
Fine Tuning: We fine tune the pre-trained model, from the previous step, over a small set of labelled question-answer pairs  As we shall later see, this step is crucial, and it only requires a handful of labelled questions to achieve a significant proportion of the performance typically attained by training on tens of thousands of questions 
Datasets
We apply our system to three datasets from different domains  SQuAD BIBREF3 consists of questions whose answers are free form spans of text from passages in Wikipedia articles  We follow the same setting as in BIBREF0 , and split $10\%$ of training questions as the test set, and report performance when training on subsets of the remaining data ranging from $1\%$ to $90\%$ of the full set  We also report the performance on the dev set when trained on the full training set ( $1^\ast $ in Table 2 )  We use the same hyperparameter settings as in prior work  We compare and study four different settings: 1) the Supervised Learning (SL) setting, which is only trained on the supervised data, 2) the best performing GDAN model from BIBREF0 , 3) pretraining on a Language Modeling (LM) objective and fine-tuning on the supervised data, and 4) pretraining on the Cloze dataset and fine-tuning on the supervised data  The LM and Cloze methods use exactly the same data for pretraining, but differ in the loss functions used  We report F1 and EM scores on our test set using the official evaluation scripts provided by the authors of the dataset 
TriviaQA BIBREF4 comprises of over 95K web question-answer-evidence triples  Like SQuAD, the answers are spans of text  Similar to the setting in SQuAD, we create multiple smaller subsets of the entire set  For our semi-supervised QA system, we use the BiDAF+SA model BIBREF1 – the highest performing publicly available system for TrivaQA  Here again, we compare the supervised learning (SL) settings against the pretraining on Cloze set and fine tuning on the supervised set  We report F1 and EM scores on the dev set 
We also test on the BioASQ 5b dataset, which consists of question-answer pairs from PubMed abstracts  We use the publicly available system from BIBREF14 , and follow the exact same setup as theirs, focusing only on factoid and list questions  For this setting, there are only 899 questions for training  Since this is already a low-resource problem we only report results using 5-fold cross-validation on all the available data  We report Mean Reciprocal Rank (MRR) on the factoid questions, and F1 score for the list questions 
Main Results
Table 2 shows a comparison of the discussed settings on both SQuAD and TriviaQA  Without any fine-tuning (column 0) the performance is low, probably because the model never saw a real question, but we see significant gains with Cloze pretraining even with very little labeled data  The BiDAF+SA model, exceeds an F1 score of $50\%$ with only $1\%$ of the training data (454 questions for SQuAD, and 746 questions for TriviaQA), and approaches $90\%$ of the best performance with only $10\%$ labeled data  The gains over the SL setting, however, diminish as the size of the labeled set increases and are small when the full dataset is available 
Cloze pretraining outperforms the GDAN baseline from BIBREF0 using the same SQuAD dataset splits  Additionally, we show improvements in the $90\%$ data case unlike GDAN  Our approach is also applicable in the extremely low-resource setting of $1\%$ data, which we suspect GDAN might have trouble with since it uses the labeled data to do reinforcement learning  Furthermore, we are able to use the same cloze dataset to improve performance on both SQuAD and TriviaQA datasets  When we use the same unlabeled data to pre-train with a language modeling objective, the performance is worse, showing the bias we introduce by constructing clozes is important 
On the BioASQ dataset (Table 3 ) we again see a significant improvement when pretraining with the cloze questions over the supervised baseline  The improvement is smaller than what we observe with SQuAD and TriviaQA datasets – we believe this is because questions are generally more difficult in BioASQ  BIBREF14 showed that pretraining on SQuAD dataset improves the downstream performance on BioASQ  Here, we show a much larger improvement by pretraining on cloze questions constructed in an unsupervised manner from the same domain 
Analysis
Regression Analysis: To understand which types of questions benefit from pre-training, we pre-specified certain features (see Figure 1 right) for each of the dev set questions in SQuAD, and then performed linear regression to predict the F1 score for that question from these features  We predict the F1 scores from the cloze pretrained model ( $y^{\text{cloze}}$ ), the supervised model ( $y^{\text{sl}}$ ), and the difference of the two ( $y^{\text{cloze}}-y^{\text{sl}}$ ), when using $10\%$ of labeled data  The coefficients of the fitted model are shown in Figure 1 (left) along with their std errors  Positive coefficients indicate that a high value of that feature is predictive of a high F1 score, and a negative coefficient indicates that a small value of that feature is predictive of a high F1 score (or a high difference of F1 scores from the two models in the case of $y^{\text{cloze}}-y^{\text{sl}}$ ) 
The two strongest effects we observe are that a high lexical overlap between the question and the sentence containing the answer is indicative of high boost with pretraining, and that a high lexical overlap between the question and the whole passage is indicative of the opposite  This is hardly surprising, since our cloze construction process is biased towards questions which have a similar phrasing to the answer sentences in context  Hence, test questions with a similar property are answered correctly after pretraining, whereas those with a high overlap with the whole passage tend to have lower performance  The pretraining also favors questions with short answers because the cloze construction process produces short answer spans  Also passages and questions which consist of tokens infrequent in the SQuAD training corpus receive a large boost after pretraining, since the unlabeled data covers a larger domain 
Performance on question types: Figure 2 shows the average gain in F1 score for different types of questions, when we pretrain on the clozes compared to the supervised case  This analysis is done on the $10\%$ split of the SQuAD training set  We consider two classifications of each question – one determined on the first word (usually a wh-word) of the question (Figure 2 (bottom)) and one based on the output of a separate question type classifier adapted from BIBREF21   We use the coarse grain labels namely Abbreviation (ABBR), Entity (ENTY), Description (DESC), Human (HUM), Location (LOC), Numeric (NUM) trained on a Logistic Regression classification system   While there is an improvement across the board, we find that abbreviation questions in particular receive a large boost  Also, "why" questions show the least improvement, which is in line with our expectation, since these usually require reasoning or world knowledge which cloze questions rarely require 
Conclusion
In this paper, we show that pre-training QA models with automatically constructed cloze questions improves the performance of the models significantly, especially when there are few labeled examples  The performance of the model trained only on the cloze questions is poor, validating the need for fine-tuning  Through regression analysis, we find that pretraining helps with questions which ask for factual information located in a specific part of the context  For future work, we plan to explore the active learning setup for this task – specifically, which passages and / or types of questions can we select to annotate, such that there is a maximum performance gain from fine-tuning  We also want to explore how to adapt cloze style pre-training to NLP tasks other than QA 
Acknowledgments
Bhuwan Dhingra is supported by NSF under grants CCF-1414030 and IIS-1250956 and by grants from Google  Danish Pruthi and Dheeraj Rajagopal are supported by the DARPA Big Mechanism program under ARO contract W911NF-14-1-0436 

ground_truth: For Article-Entity placement, they consider two baselines: the first one using only salience-based features, and the second baseline checks if the entity appears in the title of the article. 

For Article-Section Placement, they consider two baselines: the first picks the section with the highest lexical similarity to the article, and the second one picks the most frequent section.

B1. The first baseline uses only the salience-based features by Dunietz and Gillick BIBREF11 ., B2. The second baseline assigns the value relevant to a pair INLINEFORM0 , if and only if INLINEFORM1 appears in the title of INLINEFORM2 .

, S1: Pick the section from template INLINEFORM0 with the highest lexical similarity to INLINEFORM1 : S1 INLINEFORM2, S2: Place the news into the most frequent section in INLINEFORM0

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is it possible to convert a cloze-style questions to a naturally-looking questions?
Context 4:  Especially findings often appear as discontinuous entities, we found 543 discontinuous finding relations  The numbers for conditions and factors are lower with seven and two, respectively  Entities can also be nested within one another  This happens either when the span of one annotation is completely embedded in the span of another annotation (fully-nested; cf  Figure FIGREF12), or when there is a partial overlapping between the spans of two different entities (partially-nested; cf  Figure FIGREF12)  There is a high number of inter-sentential relations in the corpus (cf  Table TABREF27)  This can be explained by the fact that the case entity occurs early in each document; furthermore, it is related to finding and factor annotations that are distributed across different sentences 
The most frequently annotated relation in our corpus is the has-relation between a case entity and the findings related to that case  This correlates with the high number of finding entities  The relations contained in our corpus are summarized in Table TABREF27 
Baseline systems for Named Entity Recognition in medical case reports
We evaluate the corpus using Named Entity Recognition (NER), i  e , the task of finding mentions of concepts of interest in unstructured text  We focus on detecting cases, conditions, factors, findings and modifiers in case reports (cf  Section SECREF6)  We approach this as a sequence labeling problem  Four systems were developed to offer comparable robust baselines 
The original documents are pre-processed (sentence splitting and tokenization with ScispaCy)  We do not perform stop word removal or lower-casing of the tokens  The BIO labeling scheme is used to capture the order of tokens belonging to the same entity type and enable span-level detection of entities  Detection of nested and/or discontinuous entities is not supported  The annotated corpus is randomized and split in five folds using scikit-learn BIBREF9  Each fold has a train, test and dev split with the test split defined as  15% of the train split  This ensures comparability between the presented systems 
Baseline systems for Named Entity Recognition in medical case reports ::: Conditional Random Fields
Conditional Random Fields (CRF) BIBREF10 are a standard approach when dealing with sequential data in the context of sequence labeling  We use a combination of linguistic and semantic features, with a context window of size five, to describe each of the tokens and the dependencies between them  Hyper-parameter optimization is performed using randomized search and cross validation  Span-based F1 score is used as the optimization metric 
Baseline systems for Named Entity Recognition in medical case reports ::: BiLSTM-CRF
Prior to the emergence of deep neural language models, BiLSTM-CRF models BIBREF11 had achieved state-of-the-art results for the task of sequence labeling  We use a BiLSTM-CRF model with both word-level and character-level input  BioWordVec BIBREF12 pre-trained word embeddings are used in the embedding layer for the input representation  A bidirectional LSTM layer is applied to a multiplication of the two input representations  Finally, a CRF layer is applied to predict the sequence of labels  Dropout and L1/L2 regularization is used where applicable  He (uniform) initialization BIBREF13 is used to initialize the kernels of the individual layers  As the loss metric, CRF-based loss is used, while optimizing the model based on the CRF Viterbi accuracy  Additionally, span-based F1 score is used to serialize the best performing model  We train for a maximum of 100 epochs, or until an early stopping criterion is reached (no change in validation loss value grater than 0 01 for ten consecutive epochs)  Furthermore, Adam BIBREF14 is used as the optimizer  The learning rate is reduced by a factor of 0 3 in case no significant increase of the optimization metric is achieved in three consecutive epochs 
Baseline systems for Named Entity Recognition in medical case reports ::: Multi-Task Learning
Multi-Task Learning (MTL) BIBREF15 has become popular with the progress in deep learning  This model family is characterized by simultaneous optimization of multiple loss functions and transfer of knowledge achieved this way  The knowledge is transferred through the use of one or multiple shared layers  Through finding supporting patterns in related tasks, MTL provides better generalization on unseen cases and the main tasks we are trying to solve 
We rely on the model presented by bekoulis2018joint and reuse the implementation provided by the authors  The model jointly trains two objectives supported by the dataset: the main task of NER and a supporting task of Relation Extraction (RE)  Two separate models are developed for each of the tasks  The NER task is solved with the help of a BiLSTM-CRF model, similar to the one presented in Section SECREF32 The RE task is solved by using a multi-head selection approach, where each token can have none or more relationships to in-sentence tokens  Additionally, this model also leverages the output of the NER branch model (the CRF prediction) to learn label embeddings  Shared layers consist of a concatenation of word and character embeddings followed by two bidirectional LSTM layers  We keep most of the parameters suggested by the authors and change (1) the number of training epochs to 100 to allow the comparison to other deep learning approaches in this work, (2) use label embeddings of size 64, (3) allow gradient clipping and (4) use $d=0 8$ as the pre-trained word embedding dropout and $d=0 5$ for all other dropouts  $\eta =1^{-3}$ is used as the learning rate with the Adam optimizer and tanh activation functions across layers  Although it is possible to use adversarial training BIBREF16, we omit from using it  We also omit the publication of results for the task of RE as we consider it to be a supporting task and no other competing approaches have been developed 
Baseline systems for Named Entity Recognition in medical case reports ::: BioBERT
Deep neural language models have recently evolved to a successful method for representing text  In particular, Bidirectional Encoder Representations from Transformers (BERT) outperformed previous state-of-the-art methods by a large margin on various NLP tasks BIBREF17  For our experiments, we use BioBERT, an adaptation of BERT for the biomedical domain, pre-trained on PubMed abstracts and PMC full-text articles BIBREF18  The BERT architecture for deriving text representations uses 12 hidden layers, consisting of 768 units each  For NER, token level BIO-tag probabilities are computed with a single output layer based on the representations from the last layer of BERT  We fine-tune the model on the entity recognition task during four training epochs with batch size $b=32$, dropout probability $d=0 1$ and learning rate $\eta =2^{-5}$  These hyper-parameters are proposed by Devlin2018 for BERT fine-tuning 
Baseline systems for Named Entity Recognition in medical case reports ::: Evaluation
To evaluate the performance of the four systems, we calculate the span-level precision (P), recall (R) and F1 scores, along with corresponding micro and macro scores  The reported values are shown in Table TABREF29 and are averaged over five folds, utilising the seqeval framework 
With a macro avg  F1-score of 0 59, MTL achieves the best result with a significant margin compared to CRF, BiLSTM-CRF and BERT  This confirms the usefulness of jointly training multiple objectives (minimizing multiple loss functions), and enabling knowledge transfer, especially in a setting with limited data (which is usually the case in the biomedical NLP domain)  This result also suggest the usefulness of BioBERT for other biomedical datasets as reported by Lee2019  Despite being a rather standard approach, CRF outperforms the more elaborated BiLSTM-CRF, presumably due to data scarcity and class imbalance  We hypothesize that an increase in training data would yield better results for BiLSTM-CRF but not outperform transfer learning approach of MTL (or even BioBERT)  In contrast to other common NER corpora, like CoNLL 2003, even the best baseline system only achieves relatively low scores  This outcome is due to the inherent difficulty of the task (annotators are experienced medical doctors) and the small number of training samples 
Conclusion
We present a new corpus, developed to facilitate the processing of case reports  The corpus focuses on five distinct entity types: cases, conditions, factors, findings and modifiers  Where applicable, relationships between entities are also annotated  Additionally, we annotate discontinuous entities with a special relationship type (discontinuous)  The corpus presented in this paper is the very first of its kind and a valuable addition to the scarce number of corpora available in the field of biomedical NLP  Its complexity, given the discontinuous nature of entities and a high number of nested and multi-label entities, poses new challenges for NLP methods applied for NER and can, hence, be a valuable source for insights into what entities “look like in the wild”  Moreover, it can serve as a playground for new modelling techniques such as the resolution of discontinuous entities as well as multi-task learning given the combination of entities and their relations  We provide an evaluation of four distinct NER systems that will serve as robust baselines for future work but which are, as of yet, unable to solve all the complex challenges this dataset holds  A functional service based on the presented corpus is currently being integrated, as a NER service, in the QURATOR platform BIBREF20 
Acknowledgments
The research presented in this article is funded by the German Federal Ministry of Education and Research (BMBF) through the project QURATOR (Unternehmen Region, Wachstumskern, grant no  03WKDA1A), see http://qurator ai  We want to thank our medical experts for their help annotating the data set, especially Ashlee Finckh and Sophie Klopfenstein 

ground_truth: Unanswerable

Unanswerable