prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the combination of rewards for reinforcement learning?
Context 1: Introduction
The irony is a kind of figurative language, which is widely used on social media BIBREF0   The irony is defined as a clash between the intended meaning of a sentence and its literal meaning BIBREF1   As an important aspect of language, irony plays an essential role in sentiment analysis BIBREF2 , BIBREF0 and opinion mining BIBREF3 , BIBREF4  
Although some previous studies focus on irony detection, little attention is paid to irony generation  As ironies can strengthen sentiments and express stronger emotions, we mainly focus on generating ironic sentences  Given a non-ironic sentence, we implement a neural network to transfer it to an ironic sentence and constrain the sentiment polarity of the two sentences to be the same  For example, the input is “I hate it when my plans get ruined" which is negative in sentiment polarity and the output should be ironic and negative in sentiment as well, such as “I like it when my plans get ruined"  The speaker uses “like" to be ironic and express his or her negative sentiment  At the same time, our model can preserve contents which are irrelevant to sentiment polarity and irony  According to the categories mentioned in BIBREF5 , irony can be classified into 3 classes: verbal irony by means of a polarity contrast, the sentences containing expression whose polarity is inverted between the intended and the literal evaluation; other types of verbal irony, the sentences that show no polarity contrast between the literal and intended meaning but are still ironic; and situational irony, the sentences that describe situations that fail to meet some expectations  As ironies in the latter two categories are obscure and hard to understand, we decide to only focus on ironies in the first category in this work  For example, our work can be specifically described as: given a sentence “I hate to be ignored", we train our model to generate an ironic sentence such as “I love to be ignored"  Although there is “love" in the generated sentence, the speaker still expresses his or her negative sentiment by irony  We also make some explorations in the transformation from ironic sentences to non-ironic sentences at the end of our work  Because of the lack of previous work and baselines on irony generation, we implement our model based on style transfer  Our work will not only provide the first large-scale irony dataset but also make our model as a benchmark for the irony generation 
Recently, unsupervised style transfer becomes a very popular topic  Many state-of-the-art studies try to solve the task with sequence-to-sequence (seq2seq) framework  There are three main ways to build up models  The first is to learn a latent style-independent content representation and generate sentences with the content representation and another style BIBREF6 , BIBREF7   The second is to directly transfer sentences from one style to another under the control of classifiers and reinforcement learning BIBREF8   The third is to remove style attribute words from the input sentence and combine the remaining content with new style attribute words BIBREF9 , BIBREF10   The first method usually obtains better performances via adversarial training with discriminators  The style-independent content representation, nevertheless, is not easily obtained BIBREF11 , which results in poor performances  The second method is suitable for complex styles which are difficult to model and describe  The model can learn the deep semantic features by itself but sometimes the model is sensitive to parameters and hard to train  The third method succeeds to preserve content but cannot work for some complex styles such as democratic and republican  Sentences with those styles usually do not have specific style attribute words  Unfortunately, due to the lack of large irony dataset and difficulties of modeling ironies, there has been little work trying to generate ironies based on seq2seq framework as far as we know  Inspired by methods for style transfer, we decide to implement a specifically designed model based on unsupervised style transfer to explore irony generation 
In this paper, in order to address the lack of irony data, we first crawl over 2M tweets from twitter to build a dataset with 262,755 ironic and 112,330 non-ironic tweets  Then, due to the lack of parallel data, we propose a novel model to transfer non-ironic sentences to ironic sentences in an unsupervised way  As ironic style is hard to model and describe, we implement our model with the control of classifiers and reinforcement learning  Different from other studies in style transfer, the transformation from non-ironic to ironic sentences has to preserve sentiment polarity as mentioned above  Therefore, we not only design an irony reward to control the irony accuracy and implement denoising auto-encoder and back-translation to control content preservation but also design a sentiment reward to control sentiment preservation 
Experimental results demonstrate that our model achieves a high irony accuracy with well-preserved sentiment and content  The contributions of our work are as follows:
Related Work
Style Transfer: As irony is a complicated style and hard to model with some specific style attribute words, we mainly focus on studies without editing style attribute words 
Some studies are trying to disentangle style representation from content representation  In BIBREF12 , authors leverage adversarial networks to learn separate content representations and style representations  In BIBREF13 and BIBREF6 , researchers combine variational auto-encoders (VAEs) with style discriminators 
However, some recent studies BIBREF11 reveal that the disentanglement of content and style representations may not be achieved in practice  Therefore, some other research studies BIBREF9 , BIBREF10 strive to separate content and style by removing stylistic words  Nonetheless, many non-ironic sentences do not have specific stylistic words and as a result, we find it difficult to transfer non-ironic sentences to ironic sentences through this way in practice 
Besides, some other research studies do not disentangle style from content but directly learn representations of sentences  In BIBREF8 , authors propose a dual reinforcement learning framework without separating content and style representations  In BIBREF7 , researchers utilize a machine translation model to learn a sentence representation preserving the meaning of the sentence but reducing stylistic properties  In this method, the quality of generated sentences relies on the performance of classifiers to a large extent  Meanwhile, such models are usually sensitive to parameters and difficult to train  In contrast, we combine a pre-training process with reinforcement learning to build up a stable language model and design special rewards for our task 
Irony Detection: With the development of social media, irony detection becomes a more important task  Methods for irony detection can be mainly divided into two categories: methods based on feature engineering and methods based on neural networks 
As for methods based on feature engineering, In BIBREF1 , authors investigate pragmatic phenomena and various irony markers  In BIBREF14 , researchers leverage a combination of sentiment, distributional semantic and text surface features  Those models rely on hand-crafted features and are hard to implement 
When it comes to methods based on neural networks, long short-term memory (LSTM) BIBREF15 network is widely used and is very efficient for irony detection  In BIBREF16 , a tweet is divided into two segments and a subtract layer is implemented to calculate the difference between two segments in order to determine whether the tweet is ironic  In BIBREF17 , authors utilize a recurrent neural network with Bi-LSTM and self-attention without hand-crafted features  In BIBREF18 , researchers propose a system based on a densely connected LSTM network 
Our Dataset
In this section, we describe how we build our dataset with tweets  First, we crawl over 2M tweets from twitter using GetOldTweets-python  We crawl English tweets from 04/09/2012 to /12/18/2018  We first remove all re-tweets and use langdetect to remove all non-English sentences  Then, we remove hashtags attached at the end of the tweets because they are usually not parts of sentences and will confuse our language model  After that, we utilize Ekphrasis to process tweets  We remove URLs and restore remaining hashtags, elongated words, repeated words, and all-capitalized words  To simplify our dataset, We replace all “ INLINEFORM0 money INLINEFORM1 " and “ INLINEFORM2 time INLINEFORM3 " tokens with “ INLINEFORM4 number INLINEFORM5 " token when using Ekphrasis  And we delete sentences whose lengths are less than 10 or greater than 40  In order to restore abbreviations, we download an abbreviation dictionary from webopedia and restore abbreviations to normal words or phrases according to the dictionary  Finally, we remove sentences which have more than two rare words (appearing less than three times) in order to constrain the size of vocabulary  Finally, we get 662,530 sentences after pre-processing 
As neural networks are proved effective in irony detection, we decide to implement a neural classifier in order to classify the sentences into ironic and non-ironic sentences  However, the only high-quality irony dataset we can obtain is the dataset of Semeval-2018 Task 3 and the dataset is pretty small, which will cause overfitting to complex models  Therefore, we just implement a simple one-layer RNN with LSTM cell to classify pre-processed sentences into ironic sentences and non-ironic sentences because LSTM networks are widely used in irony detection  We train the model with the dataset of Semeval-2018 Task 3  After classification, we get 262,755 ironic sentences and 399,775 non-ironic sentences  According to our observation, not all non-ironic sentences are suitable to be transferred into ironic sentences  For example, “just hanging out   watching   is it monday yet" is hard to transfer because it does not have an explicit sentiment polarity  So we remove all interrogative sentences from the non-ironic sentences and only obtain the sentences which have words expressing strong sentiments  We evaluate the sentiment polarity of each word with TextBlob and we view those words with sentiment scores greater than 0 5 or less than -0 5 as words expressing strong sentiments  Finally, we build our irony dataset with 262,755 ironic sentences and 102,330 non-ironic sentences 
[t] Irony Generation Algorithm
INLINEFORM0 pre-train with auto-encoder Pre-train INLINEFORM1 , INLINEFORM2 with INLINEFORM3 using MLE based on Eq  EQREF16 Pre-train INLINEFORM4 , INLINEFORM5 with INLINEFORM6 using MLE based on Eq  EQREF17 INLINEFORM7 pre-train with back-translation Pre-train INLINEFORM8 , INLINEFORM9 , INLINEFORM10 , INLINEFORM11 with INLINEFORM12 using MLE based on Eq  EQREF19 Pre-train INLINEFORM13 , INLINEFORM14 , INLINEFORM15 , INLINEFORM16 with INLINEFORM17 using MLE based on Eq  EQREF20
INLINEFORM0 train with RL each epoch e = 1, 2,    , INLINEFORM1 INLINEFORM2 train non-irony2irony with RL INLINEFORM3 in N INLINEFORM4 update INLINEFORM5 , INLINEFORM6 , using INLINEFORM7 based on Eq  EQREF29 INLINEFORM8 back-translation INLINEFORM9 INLINEFORM10 INLINEFORM11 update INLINEFORM12 , INLINEFORM13 , INLINEFORM14 , INLINEFORM15 using MLE based on Eq  EQREF19 INLINEFORM16 train irony2non-irony with RL INLINEFORM17 in I INLINEFORM18 update INLINEFORM19 , INLINEFORM20 , using INLINEFORM21 similar to Eq  EQREF29 INLINEFORM22 back-translation INLINEFORM23 INLINEFORM24 INLINEFORM25 update INLINEFORM26 , INLINEFORM27 , INLINEFORM28 , INLINEFORM29 using MLE based on Eq  EQREF20
Our Method
Given two non-parallel corpora: non-ironic corpus N={ INLINEFORM0 , INLINEFORM1 ,    , INLINEFORM2 } and ironic corpus I={ INLINEFORM3 , INLINEFORM4 ,    , INLINEFORM5 }, the goal of our irony generation model is to generate an ironic sentence from a non-ironic sentence while preserving the content and sentiment polarity of the source input sentence  We implement an encoder-decoder framework where two encoders are utilized to encode ironic sentences and non-ironic sentences respectively and two decoders are utilized to decode ironic sentences and non-ironic sentences from latent representations respectively  In order to enforce a shared latent space, we share two layers on both the encoder side and the decoder side  Our model architecture is illustrated in Figure FIGREF13   We denote irony encoder as INLINEFORM6 , irony decoder as INLINEFORM7 and non-irony encoder as INLINEFORM8 , non-irony decoder as INLINEFORM9   Their parameters are INLINEFORM10 , INLINEFORM11 , INLINEFORM12 and INLINEFORM13  
Our irony generation algorithm is shown in Algorithm SECREF3   We first pre-train our model using denoising auto-encoder and back-translation to build up language models for both styles (section SECREF14 )  Then we implement reinforcement learning to train the model to transfer sentences from one style to another (section SECREF21 )  Meanwhile, to achieve content preservation, we utilize back-translation for one time in every INLINEFORM0 time steps 
Pretraining
In order to build up our language model and preserve the content, we apply the auto-encoder model  To prevent the model from simply copying the input sentence, we randomly add some noises in the input sentence  Specifically, for every word in the input sentence, there is 10% chance that we delete it, 10 % chance that we duplicate it, 10% chance that we swap it with the next word, or it remains unchanged  We first encode the input sentence INLINEFORM0 or INLINEFORM1 with respective encoder INLINEFORM2 or INLINEFORM3 to obtain its latent representation INLINEFORM4 or INLINEFORM5 and reconstruct the input sentence with the latent representation and respective decoder  So we can get the reconstruction loss for auto-encoder INLINEFORM6 : DISPLAYFORM0 DISPLAYFORM1
In addition to denoising auto-encoder, we implement back-translation BIBREF19 to generate a pseudo-parallel corpus  Suppose our model takes non-ironic sentence INLINEFORM0 as input  We first encode INLINEFORM1 with INLINEFORM2 to obtain its latent representation INLINEFORM3 and decode the latent representation with INLINEFORM4 to get a transferred sentence INLINEFORM5   Then we encode INLINEFORM6 with INLINEFORM7 and decode its latent representation with INLINEFORM8 to reconstruct the original input sentence INLINEFORM9   Therefore, our reconstruction loss for back-translation INLINEFORM10 : DISPLAYFORM0
And if our model takes ironic sentence INLINEFORM0 as input, we can get the reconstruction loss for back-translation as: DISPLAYFORM0
Reinforcement Learning
Since the gold transferred result of input is unavailable, we cannot evaluate the quality of the generated sentence directly  Therefore, we implement reinforcement learning and elaborately design two rewards to describe the irony accuracy and sentiment preservation, respectively 
A pre-trained binary irony classifier based on CNN BIBREF20 is used to evaluate how ironic a sentence is  We denote the parameter of the classifier as INLINEFORM0 and it is fixed during the training process 
In order to facilitate the transformation, we design the irony reward as the difference between the irony score of the input sentence and that of the output sentence  Formally, when we input a non-ironic sentence INLINEFORM0 and transfer it to an ironic sentence INLINEFORM1 , our irony reward is defined as: DISPLAYFORM0
where INLINEFORM0 denotes ironic style and INLINEFORM1 is the probability of that a sentence INLINEFORM2 is ironic 
To preserve the sentiment polarity of the input sentence, we also need to use classifiers to evaluate the sentiment polarity of the sentences  However, the sentiment analysis of ironic sentences and non-ironic sentences are different  In the case of figurative languages such as irony, sarcasm or metaphor, the sentiment polarity of the literal meaning may differ significantly from that of the intended figurative meaning BIBREF0   As we aim to train our model to transfer sentences from non-ironic to ironic, using only one classifier is not enough  As a result, we implement two pre-trained sentiment classifiers for non-ironic sentences and ironic sentences respectively  We denote the parameter of the sentiment classifier for ironic sentences as INLINEFORM0 and that of the sentiment classifier for non-ironic sentences as INLINEFORM1  
A challenge, when we implement two classifiers to evaluate the sentiment polarity, is that the two classifiers trained with different datasets may have different distributions of scores  That means we cannot directly calculate the sentiment reward with scores applied by two classifiers  To alleviate this problem and standardize the prediction results of two classifiers, we set a threshold for each classifier and subtract the respective threshold from scores applied by the classifier to obtain the comparative sentiment polarity score  We get the optimal threshold by maximizing the ability of the classifier according to the distribution of our training data 
We denote the threshold of ironic sentiment classifier as INLINEFORM0 and the threshold of non-ironic sentiment classifier as INLINEFORM1   The standardized sentiment score is defined as INLINEFORM2 and INLINEFORM3 where INLINEFORM4 denotes the positive sentiment polarity and INLINEFORM5 is the probability of that a sentence is positive in sentiment polarity 
As mentioned above, the input sentence and the generated sentence should express the same sentiment  For example, if we input a non-ironic sentence “I hate to be ignored" which is negative in sentiment polarity, the generated ironic sentence should be also negative, such as “I love to be ignored"  To achieve sentiment preservation, we design the sentiment reward as that one minus the absolute value of the difference between the standardized sentiment score of the input sentence and that of the generated sentence  Formally, when we input a non-ironic sentence INLINEFORM0 and transfer it to an ironic sentence INLINEFORM1 , our sentiment reward is defined as: DISPLAYFORM0
To encourage our model to focus on both the irony accuracy and the sentiment preservation, we apply the harmonic mean of irony reward and sentiment reward: DISPLAYFORM0
Policy Gradient
The policy gradient algorithm BIBREF21 is a simple but widely-used algorithm in reinforcement learning  It is used to maximize the expected reward INLINEFORM0   The objective function to minimize is defined as: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 is the reward of INLINEFORM2 and INLINEFORM3 is the input size 
Training Details
INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 in our model are Transformers BIBREF22 with 4 layers and 2 shared layers  The word embeddings of 128 dimensions are learned during the training process  Our maximum sentence length is set as 40  The optimizer is Adam BIBREF23 and the learning rate is INLINEFORM4   The batch size is 32 and harmonic weight INLINEFORM5 in Eq 9 is 0 5  We set the interval INLINEFORM6 as 200  The model is pre-trained for 6 epochs and trained for 15 epochs for reinforcement learning 
Irony Classifier: We implement a CNN classifier trained with our irony dataset  All the CNN classifiers we utilize in this paper use the same parameters as BIBREF20  
Sentiment Classifier for Irony: We first implement a one-layer LSTM network to classify ironic sentences in our dataset into positive and negative ironies  The LSTM network is trained with the dataset of Semeval 2015 Task 11 BIBREF0 which is used for the sentiment analysis of figurative language in twitter  Then, we use the positive ironies and negative ironies to train the CNN sentiment classifier for irony 
Sentiment Classifier for Non-irony: Similar to the training process of the sentiment classifier for irony, we first implement a one-layer LSTM network trained with the dataset for the sentiment analysis of common twitters to classify the non-ironies into positive and negative non-ironies  Then we use the positive and negative non-ironies to train the sentiment classifier for non-irony 
Baselines
We compare our model with the following state-of-art generative models:
BackTrans BIBREF7 : In BIBREF7 , authors propose a model using machine translation in order to preserve the meaning of the sentence while reducing stylistic properties 
Unpaired BIBREF10 : In BIBREF10 , researchers implement a method to remove emotional words and add desired sentiment controlled by reinforcement learning 
CrossAlign BIBREF6 : In BIBREF6 , authors leverage refined alignment of latent representations to perform style transfer and a cross-aligned auto-encoder is implemented 
CPTG BIBREF24 : An interpolated reconstruction loss is introduced in BIBREF24 and a discriminator is implemented to control attributes in this work 
DualRL BIBREF8 : In BIBREF8 , researchers use two reinforcement rewards simultaneously to control style accuracy and content preservation 
Evaluation Metrics
In order to evaluate sentiment preservation, we use the absolute value of the difference between the standardized sentiment score of the input sentence and that of the generated sentence  We call the value as sentiment delta (senti delta)  Besides, we report the sentiment accuracy (Senti ACC) which measures whether the output sentence has the same sentiment polarity as the input sentence based on our standardized sentiment classifiers  The BLEU score BIBREF25 between the input sentences and the output sentences is calculated to evaluate the content preservation performance  In order to evaluate the overall performance of different models, we also report the geometric mean (G2) and harmonic mean (H2) of the sentiment accuracy and the BLEU score  As for the irony accuracy, we only report it in human evaluation results because it is more accurate for the human to evaluate the quality of irony as it is very complicated 
We first sample 50 non-ironic input sentences and their corresponding output sentences of different models  Then, we ask four annotators who are proficient in English to evaluate the qualities of the generated sentences of different models  They are required to rank the output sentences of our model and baselines from the best to the worst in terms of irony accuracy (Irony), Sentiment preservation (Senti) and content preservation (Content)  The best output is ranked with 1 and the worst output is ranked with 6  That means that the smaller our human evaluation value is, the better the corresponding model is 
Results and Discussions
Table TABREF35 shows the automatic evaluation results of the models in the transformation from non-ironic sentences to ironic sentences  From the results, our model obtains the best result in sentiment delta  The DualRL model achieves the highest result in other metrics, but most of its outputs are the almost same as the input sentences  So it is reasonable that DualRL system outperforms ours in these metrics but it actually does not transfer the non-ironic sentences to ironic sentences at all  From this perspective, we cannot view DualRL as an effective model for irony generation  In contrast, our model gets results close to those of DualRL and obtains a balance between irony accuracy, sentiment preservation, and content preservation if we also consider the irony accuracy discussed below 
And from human evaluation results shown in Table TABREF36 , our model gets the best average rank in irony accuracy  And as mentioned above, the DualRL model usually does not change the input sentence and outputs the same sentence  Therefore, it is reasonable that it obtains the best rank in sentiment and content preservation and ours is the second  However, it still demonstrates that our model, instead of changing nothing, transfers the style of the input sentence with content and sentiment preservation at the same time 
Case Study
In the section, we present some example outputs of different models  Table TABREF37 shows the results of the transformation from non-ironic sentences to ironic sentences  We can observe that: (1) The BackTrans system, the Unpaired system, the CrossAlign system and the CPTG system tends to generate sentences which are towards irony but do not preserve content  (2) The DualRL system preserves content and sentiment very well but even does not change the input sentence  (3) Our model considers both aspects and achieves a better balance among irony accuracy, sentiment and content preservation 
Error Analysis
Although our model outperforms other style transfer baselines according to automatic and human evaluation results, there are still some failure cases because irony generation is still a very challenging task  We would like to share the issues we meet during our experiments and our solutions to some of them in this section 
No Change: As mentioned above, many style transfer models, such as DualRL, tend to make few changes to the input sentence and output the same sentence  Actually, this is a common issue for unsupervised style transfer systems and we also meet it during our experiments  The main reason for the issue is that rewards for content preservation are too prominent and rewards for style accuracy cannot work well  In contrast, in order to guarantee the readability and fluency of the output sentence, we also cannot emphasize too much on rewards for style accuracy because it may cause some other issues such as word repetition mentioned below  A method to solve the problem is tuning hyperparameters and this is also the method we implement in this work  As for content preservation, maybe MLE methods such as back-translation are not enough because they tend to force models to generate specific words  In the future, we should further design some more suitable methods to control content preservation for models without disentangling style and content representations, such as DualRL and ours 
Word Repetition: During our experiments, we observe that some of the outputs prefer to repeat the same word as shown in Table TABREF38   This is because reinforcement learning rewards encourage the model to generate words which can get high scores from classifiers and even back-translation cannot stop it  Our solution is that we can lower the probability of decoding a word in decoders if the word has been generated in the previous time steps during testing  We also try to implement this method during training time but obtain worse performances because it may limit the effects of training  Some previous studies utilize language models to control the fluency of the output sentence and we also try this method  Nonetheless, pre-training a language model with tweets and using it to generate rewards is difficult because tweets are more casual and have more noise  Rewards from that kind of language model are usually not accurate and may confuse the model  In the future, we should come up with better methods to model language fluency with the consideration of irony accuracy, sentiment and content preservation, especially for tweets 
Improper Words: As ironic style is hard for our model to learn, it may generate some improper words which make the sentence strange  As the example shown in the Table TABREF38 , the sentiment word in the input sentence is “wonderful" and the model should change it into a negative word such as “sad" to make the output sentence ironic  However, the model changes “friday" and “fifa" which are not related to ironic styles  We have not found a very effective method to address this issue and maybe we should further explore stronger models to learn ironic styles better 
Additional Experiments
In this section, we describe some additional experiments on the transformation from ironic sentences to non-ironic sentences  Sometimes ironies are hard to understand and may cause misunderstanding, for which our task also explores the transformation from ironic sentences to non-ironic sentences 
As shown in Table TABREF46 , we also conduct automatic evaluations and the conclusions are similar to those of the transformation from non-ironic sentences to ironic sentences  As for human evaluation results in Table TABREF47 , our model still can achieve the second-best results in sentiment and content preservation  Nevertheless, DualRL system and ours get poor performances in irony accuracy  The reason may be that the other four baselines tend to generate common and even not fluent sentences which are irrelevant to the input sentences and are hard to be identified as ironies  So annotators usually mark these output sentences as non-ironic sentences, which causes these models to obtain better performances than DualRL and ours but much poorer results in sentiment and content preservation  Some examples are shown in Table TABREF52  
Conclusion and Future Work
In this paper, we first systematically define irony generation based on style transfer  Because of the lack of irony data, we make use of twitter and build a large-scale dataset  In order to control irony accuracy, sentiment preservation and content preservation at the same time, we also design a combination of rewards for reinforcement learning and incorporate reinforcement learning with a pre-training process  Experimental results demonstrate that our model outperforms other generative models and our rewards are effective  Although our model design is effective, there are still many errors and we systematically analyze them  In the future, we are interested in exploring these directions and our work may extend to other kinds of ironies which are more difficult to model 
Context 2: Introduction
Wikipedia is the largest source of open and collaboratively curated knowledge in the world  Introduced in 2001, it has evolved into a reference work with around 5m pages for the English Wikipedia alone  In addition, entities and event pages are updated quickly via collaborative editing and all edits are encouraged to include source citations, creating a knowledge base which aims at being both timely as well as authoritative  As a result, it has become the preferred source of information consumption about entities and events  Moreso, this knowledge is harvested and utilized in building knowledge bases like YAGO BIBREF0 and DBpedia BIBREF1 , and used in applications like text categorization BIBREF2 , entity disambiguation BIBREF3 , entity ranking BIBREF4 and distant supervision BIBREF5 , BIBREF6  
However, not all Wikipedia pages referring to entities (entity pages) are comprehensive: relevant information can either be missing or added with a delay  Consider the city of New Orleans and the state of Odisha which were severely affected by cyclones Hurricane Katrina and Odisha Cyclone, respectively  While Katrina finds extensive mention in the entity page for New Orleans, Odisha Cyclone which has 5 times more human casualties (cf  Figure FIGREF2 ) is not mentioned in the page for Odisha  Arguably Katrina and New Orleans are more popular entities, but Odisha Cyclone was also reported extensively in national and international news outlets  This highlights the lack of important facts in trunk and long-tail entity pages, even in the presence of relevant sources  In addition, previous studies have shown that there is an inherent delay or lag when facts are added to entity pages BIBREF7  
To remedy these problems, it is important to identify information sources that contain novel and salient facts to a given entity page  However, not all information sources are equal  The online presence of major news outlets is an authoritative source due to active editorial control and their articles are also a timely container of facts  In addition, their use is in line with current Wikipedia editing practice, as is shown in BIBREF7 that almost 20% of current citations in all entity pages are news articles  We therefore propose news suggestion as a novel task that enhances entity pages and reduces delay while keeping its pages authoritative 
Existing efforts to populate Wikipedia BIBREF8 start from an entity page and then generate candidate documents about this entity using an external search engine (and then post-process them)  However, such an approach lacks in (a) reproducibility since rankings vary with time with obvious bias to recent news (b) maintainability since document acquisition for each entity has to be periodically performed  To this effect, our news suggestion considers a news article as input, and determines if it is valuable for Wikipedia  Specifically, given an input news article INLINEFORM0 and a state of Wikipedia, the news suggestion problem identifies the entities mentioned in INLINEFORM1 whose entity pages can improve upon suggesting INLINEFORM2   Most of the works on knowledge base acceleration BIBREF9 , BIBREF10 , BIBREF11 , or Wikipedia page generation BIBREF8 rely on high quality input sources which are then utilized to extract textual facts for Wikipedia page population  In this work, we do not suggest snippets or paraphrases but rather entire articles which have a high potential importance for entity pages  These suggested news articles could be consequently used for extraction, summarization or population either manually or automatically – all of which rely on high quality and relevant input sources 
We identify four properties of good news recommendations: salience, relative authority, novelty and placement  First, we need to identify the most salient entities in a news article  This is done to avoid pollution of entity pages with only marginally related news  Second, we need to determine whether the news is important to the entity as only the most relevant news should be added to a precise reference work  To do this, we compute the relative authority of all entities in the news article: we call an entity more authoritative than another if it is more popular or noteworthy in the real world  Entities with very high authority have many news items associated with them and only the most relevant of these should be included in Wikipedia whereas for entities of lower authority the threshold for inclusion of a news article will be lower  Third, a good recommendation should be able to identify novel news by minimizing redundancy coming from multiple news articles  Finally, addition of facts is facilitated if the recommendations are fine-grained, i e , recommendations are made on the section level rather than the page level (placement) 
Approach and Contributions  We propose a two-stage news suggestion approach to entity pages  In the first stage, we determine whether a news article should be suggested for an entity, based on the entity's salience in the news article, its relative authority and the novelty of the article to the entity page  The second stage takes into account the class of the entity for which the news is suggested and constructs section templates from entities of the same class  The generation of such templates has the advantage of suggesting and expanding entity pages that do not have a complete section structure in Wikipedia, explicitly addressing long-tail and trunk entities  Afterwards, based on the constructed template our method determines the best fit for the news article with one of the sections 
We evaluate the proposed approach on a news corpus consisting of 351,982 articles crawled from the news external references in Wikipedia from 73,734 entity pages  Given the Wikipedia snapshot at a given year (in our case [2009-2014]), we suggest news articles that might be cited in the coming years  The existing news references in the entity pages along with their reference date act as our ground-truth to evaluate our approach  In summary, we make the following contributions 
Related Work
As we suggest a new problem there is no current work addressing exactly the same task  However, our task has similarities to Wikipedia page generation and knowledge base acceleration  In addition, we take inspiration from Natural Language Processing (NLP) methods for salience detection 
Wikipedia Page Generation is the problem of populating Wikipedia pages with content coming from external sources  Sauper and Barzilay BIBREF8 propose an approach for automatically generating whole entity pages for specific entity classes  The approach is trained on already-populated entity pages of a given class (e g  `Diseases') by learning templates about the entity page structure (e g  diseases have a treatment section)  For a new entity page, first, they extract documents via Web search using the entity title and the section title as a query, for example `Lung Cancer'+`Treatment'  As already discussed in the introduction, this has problems with reproducibility and maintainability  However, their main focus is on identifying the best paragraphs extracted from the collected documents  They rank the paragraphs via an optimized supervised perceptron model for finding the most representative paragraph that is the least similar to paragraphs in other sections  This paragraph is then included in the newly generated entity page  Taneva and Weikum BIBREF12 propose an approach that constructs short summaries for the long tail  The summaries are called `gems' and the size of a `gem' can be user defined  They focus on generating summaries that are novel and diverse  However, they do not consider any structure of entities, which is present in Wikipedia 
In contrast to BIBREF8 and BIBREF12 , we actually focus on suggesting entire documents to Wikipedia entity pages  These are authoritative documents (news), which are highly relevant for the entity, novel for the entity and in which the entity is salient  Whereas relevance in Sauper and Barzilay is implicitly computed by web page ranking we solve that problem by looking at relative authority and salience of an entity, using the news article and entity page only  As Sauper and Barzilay concentrate on empty entity pages, the problem of novelty of their content is not an issue in their work whereas it is in our case which focuses more on updating entities  Updating entities will be more and more important the bigger an existing reference work is  Both the approaches in BIBREF8 and BIBREF12 (finding paragraphs and summarization) could then be used to process the documents we suggest further  Our concentration on news is also novel 
Knowledge Base Acceleration  In this task, given specific information extraction templates, a given corpus is analyzed in order to find worthwhile mentions of an entity or snippets that match the templates  Balog BIBREF9 , BIBREF10 recommend news citations for an entity  Prior to that, the news articles are classified for their appropriateness for an entity, where as features for the classification task they use entity, document, entity-document and temporal features  The best performing features are those that measure similarity between an entity and the news document  West et al  BIBREF13 consider the problem of knowledge base completion, through question answering and complete missing facts in Freebase based on templates, i e  Frank_Zappa bornIn Baltymore, Maryland 
In contrast, we do not extract facts for pre-defined templates but rather suggest news articles based on their relevance to an entity  In cases of long-tail entities, we can suggest to add a novel section through our abstraction and generation of section templates at entity class level 
Entity Salience  Determining which entities are prominent or salient in a given text has a long history in NLP, sparked by the linguistic theory of Centering BIBREF14   Salience has been used in pronoun and co-reference resolution BIBREF15 , or to predict which entities will be included in an abstract of an article BIBREF11   Frequent features to measure salience include the frequency of an entity in a document, positioning of an entity, grammatical function or internal entity structure (POS tags, head nouns etc )  These approaches are not currently aimed at knowledge base generation or Wikipedia coverage extension but we postulate that an entity's salience in a news article is a prerequisite to the news article being relevant enough to be included in an entity page  We therefore use the salience features in BIBREF11 as part of our model  However, these features are document-internal — we will show that they are not sufficient to predict news inclusion into an entity page and add features of entity authority, news authority and novelty that measure the relations between several entities, between entity and news article as well as between several competing news articles 
Terminology and Problem Definition
We are interested in named entities mentioned in documents  An entity INLINEFORM0 can be identified by a canonical name, and can be mentioned differently in text via different surface forms  We canonicalize these mentions to entity pages in Wikipedia, a method typically known as entity linking  We denote the set of canonicalized entities extracted and linked from a news article INLINEFORM1 as INLINEFORM2   For example, in Figure FIGREF7 , entities are canonicalized into Wikipedia entity pages (e g  Odisha is canonicalized to the corresponding article)  For a collection of news articles INLINEFORM3 , we further denote the resulting set of entities by INLINEFORM4  
Information in an entity page is organized into sections and evolves with time as more content is added  We refer to the state of Wikipedia at a time INLINEFORM0 as INLINEFORM1 and the set of sections for an entity page INLINEFORM2 as its entity profile INLINEFORM3   Unlike news articles, text in Wikipedia could be explicitly linked to entity pages through anchors  The set of entities explicitly referred in text from section INLINEFORM4 is defined as INLINEFORM5   Furthermore, Wikipedia induces a category structure over its entities, which is exploited by knowledge bases like YAGO (e g  Barack_Obama isA Person)  Consequently, each entity page belongs to one or more entity categories or classes INLINEFORM6   Now we can define our news suggestion problem below:
Definition 1 (News Suggestion Problem) Given a set of news articles INLINEFORM0 and set of Wikipedia entity pages INLINEFORM1 (from INLINEFORM2 ) we intend to suggest a news article INLINEFORM3 published at time INLINEFORM4 to entity page INLINEFORM5 and additionally to the most relevant section for the entity page INLINEFORM6  
Approach Overview
We approach the news suggestion problem by decomposing it into two tasks:
AEP: Article–Entity placement
ASP: Article–Section placement
In this first step, for a given entity-news pair INLINEFORM0 , we determine whether the given news article INLINEFORM1 should be suggested (we will refer to this as `relevant') to entity INLINEFORM2   To generate such INLINEFORM3 pairs, we perform the entity linking process, INLINEFORM4 , for INLINEFORM5  
The article–entity placement task (described in detail in Section SECREF16 ) for a pair INLINEFORM0 outputs a binary label (either `non-relevant' or `relevant') and is formalized in Equation EQREF14   DISPLAYFORM0
In the second step, we take into account all `relevant' pairs INLINEFORM0 and find the correct section for article INLINEFORM1 in entity INLINEFORM2 , respectively its profile INLINEFORM3 (see Section SECREF30 )  The article–section placement task, determines the correct section for the triple INLINEFORM4 , and is formalized in Equation EQREF15   DISPLAYFORM0
In the subsequent sections we describe in details how we approach the two tasks for suggesting news articles to entity pages 
News Article Suggestion
In this section, we provide an overview of the news suggestion approach to Wikipedia entity pages (see Figure FIGREF7 )  The approach is split into two tasks: (i) article-entity (AEP) and (ii) article-section (ASP) placement  For a Wikipedia snapshot INLINEFORM0 and a news corpus INLINEFORM1 , we first determine which news articles should be suggested to an entity INLINEFORM2   We will denote our approach for AEP by INLINEFORM3   Finally, we determine the most appropriate section for the ASP task and we denote our approach with INLINEFORM4  
In the following, we describe the process of learning the functions INLINEFORM0 and INLINEFORM1   We introduce features for the learning process, which encode information regarding the entity salience, relative authority and novelty in the case of AEP task  For the ASP task, we measure the overall fit of an article to the entity sections, with the entity being an input from AEP task  Additionally, considering that the entity profiles INLINEFORM2 are incomplete, in the case of a missing section we suggest and expand the entity profiles based on section templates generated from entities of the same class INLINEFORM3 (see Section UID34 ) 
Article–Entity Placement
In this step we learn the function INLINEFORM0 to correctly determine whether INLINEFORM1 should be suggested for INLINEFORM2 , basically a binary classification model (0=`non-relevant' and 1=`relevant')  Note that we are mainly interested in finding the relevant pairs in this task  For every news article, the number of disambiguated entities is around 30 (but INLINEFORM3 is suggested for only two of them on average)  Therefore, the distribution of `non-relevant' and `relevant' pairs is skewed towards the earlier, and by simply choosing the `non-relevant' label we can achieve a high accuracy for INLINEFORM4   Finding the relevant pairs is therefore a considerable challenge 
An article INLINEFORM0 is suggested to INLINEFORM1 by our function INLINEFORM2 if it fulfills the following properties  The entity INLINEFORM3 is salient in INLINEFORM4 (a central concept), therefore ensuring that INLINEFORM5 is about INLINEFORM6 and that INLINEFORM7 is important for INLINEFORM8   Next, given the fact there might be many articles in which INLINEFORM9 is salient, we also look at the reverse property, namely whether INLINEFORM10 is important for INLINEFORM11   We do this by comparing the authority of INLINEFORM12 (which is a measure of popularity of an entity, such as its frequency of mention in a whole corpus) with the authority of its co-occurring entities in INLINEFORM13 , leading to a feature we call relative authority  The intuition is that for an entity that has overall lower authority than its co-occurring entities, a news article is more easily of importance  Finally, if the article we are about to suggest is already covered in the entity profile INLINEFORM14 , we do not wish to suggest redundant information, hence the novelty  Therefore, the learning objective of INLINEFORM15 should fulfill the following properties  Table TABREF21 shows a summary of the computed features for INLINEFORM16  
Salience: entity INLINEFORM0 should be a salient entity in news article INLINEFORM1
Relative Authority: the set of entities INLINEFORM0 with which INLINEFORM1 co-occurs should have higher authority than INLINEFORM2 , making INLINEFORM3 important for INLINEFORM4
Novelty: news article INLINEFORM0 should provide novel information for entity INLINEFORM1 taking into account its profile INLINEFORM2
Baseline Features  As discussed in Section SECREF2 , a variety of features that measure salience of an entity in text are available from the NLP community  We reimplemented the ones in Dunietz and Gillick BIBREF11   This includes a variety of features, e g  positional features, occurrence frequency and the internal POS structure of the entity and the sentence it occurs in  Table 2 in BIBREF11 gives details 
Relative Entity Frequency  Although frequency of mention and positional features play some role in baseline features, their interaction is not modeled by a single feature nor do the positional features encode more than sentence position  We therefore suggest a novel feature called relative entity frequency, INLINEFORM0 , that has three properties : (i) It rewards entities for occurring throughout the text instead of only in some parts of the text, measured by the number of paragraphs it occurs in (ii) it rewards entities that occur more frequently in the opening paragraphs of an article as we model INLINEFORM1 as an exponential decay function  The decay corresponds to the positional index of the news paragraph  This is inspired by the news-specific discourse structure that tends to give short summaries of the most important facts and entities in the opening paragraphs  (iii) it compares entity frequency to the frequency of its co-occurring mentions as the weight of an entity appearing in a specific paragraph, normalized by the sum of the frequencies of other entities in INLINEFORM2   DISPLAYFORM0
where, INLINEFORM0 represents a news paragraph from INLINEFORM1 , and with INLINEFORM2 we indicate the set of all paragraphs in INLINEFORM3   The frequency of INLINEFORM4 in a paragraph INLINEFORM5 is denoted by INLINEFORM6   With INLINEFORM7 and INLINEFORM8 we indicate the number of paragraphs in which entity INLINEFORM9 occurs, and the total number of paragraphs, respectively 
Relative Authority  In this case, we consider the comparative relevance of the news article to the different entities occurring in it  As an example, let us consider the meeting of the Sudanese bishop Elias Taban with Hillary Clinton  Both entities are salient for the meeting  However, in Taban's Wikipedia page, this meeting is discussed prominently with a corresponding news reference, whereas in Hillary Clinton's Wikipedia page it is not reported at all  We believe this is not just an omission in Clinton's page but mirrors the fact that for the lesser known Taban the meeting is big news whereas for the more famous Clinton these kind of meetings are a regular occurrence, not all of which can be reported in what is supposed to be a selection of the most important events for her  Therefore, if two entities co-occur, the news is more relevant for the entity with the lower a priori authority 
The a priori authority of an entity (denoted by INLINEFORM0 ) can be measured in several ways  We opt for two approaches: (i) probability of entity INLINEFORM1 occurring in the corpus INLINEFORM2 , and (ii) authority assessed through centrality measures like PageRank BIBREF16   For the second case we construct the graph INLINEFORM3 consisting of entities in INLINEFORM4 and news articles in INLINEFORM5 as vertices  The edges are established between INLINEFORM6 and entities in INLINEFORM7 , that is INLINEFORM8 , and the out-links from INLINEFORM9 , that is INLINEFORM10 (arrows present the edge direction) 
Starting from a priori authority, we proceed to relative authority by comparing the a priori authority of co-occurring entities in INLINEFORM0   We define the relative authority of INLINEFORM1 as the proportion of co-occurring entities INLINEFORM2 that have a higher a priori authority than INLINEFORM3 (see Equation EQREF28   DISPLAYFORM0
As we might run the danger of not suggesting any news articles for entities with very high a priori authority (such as Clinton) due to the strict inequality constraint, we can relax the constraint such that the authority of co-occurring entities is above a certain threshold 
News Domain Authority  The news domain authority addresses two main aspects  Firstly, if bundled together with the relative authority feature, we can ensure that dependent on the entity authority, we suggest news from authoritative sources, hence ensuring the quality of suggested articles  The second aspect is in a news streaming scenario where multiple news domains report the same event — ideally only articles coming from authoritative sources would fulfill the conditions for the news suggestion task 
The news domain authority is computed based on the number of news references in Wikipedia coming from a particular news domain INLINEFORM0   This represents a simple prior that a news article INLINEFORM1 is from domain INLINEFORM2 in corpus INLINEFORM3   We extract the domains by taking the base URLs from the news article URLs 
An important feature when suggesting an article INLINEFORM0 to an entity INLINEFORM1 is the novelty of INLINEFORM2 w r t the already existing entity profile INLINEFORM3   Studies BIBREF17 have shown that on comparable collections to ours (TREC GOV2) the number of duplicates can go up to INLINEFORM4   This figure is likely higher for major events concerning highly authoritative entities on which all news media will report 
Given an entity INLINEFORM0 and the already added news references INLINEFORM1 up to year INLINEFORM2 , the novelty of INLINEFORM3 at year INLINEFORM4 is measured by the KL divergence between the language model of INLINEFORM5 and articles in INLINEFORM6   We combine this measure with the entity overlap of INLINEFORM7 and INLINEFORM8   The novelty value of INLINEFORM9 is given by the minimal divergence value  Low scores indicate low novelty for the entity profile INLINEFORM10  
N(n|e) = n'Nt-1{DKL((n') || (n)) + DKL((N) || (n)) 
DKL((n') || (n))  (1-) jaccard((n'),(n))} where INLINEFORM0 is the KL divergence of the language models ( INLINEFORM1 and INLINEFORM2 ), whereas INLINEFORM3 is the mixing weight ( INLINEFORM4 ) between the language models INLINEFORM5 and the entity overlap in INLINEFORM6 and INLINEFORM7  
Here we introduce the evaluation setup and analyze the results for the article–entity (AEP) placement task  We only report the evaluation metrics for the `relevant' news-entity pairs  A detailed explanation on why we focus on the `relevant' pairs is provided in Section SECREF16  
Baselines  We consider the following baselines for this task 
B1  The first baseline uses only the salience-based features by Dunietz and Gillick BIBREF11  
B2  The second baseline assigns the value relevant to a pair INLINEFORM0 , if and only if INLINEFORM1 appears in the title of INLINEFORM2  
Learning Models  We use Random Forests (RF) BIBREF23   We learn the RF on all computed features in Table TABREF21   The optimization on RF is done by splitting the feature space into multiple trees that are considered as ensemble classifiers  Consequently, for each classifier it computes the margin function as a measure of the average count of predicting the correct class in contrast to any other class  The higher the margin score the more robust the model 
Metrics  We compute precision P, recall R and F1 score for the relevant class  For example, precision is the number of news-entity pairs we correctly labeled as relevant compared to our ground truth divided by the number of all news-entity pairs we labeled as relevant 
The following results measure the effectiveness of our approach in three main aspects: (i) overall performance of INLINEFORM0 and comparison to baselines, (ii) robustness across the years, and (iii) optimal model for the AEP placement task 
Performance  Figure FIGREF55 shows the results for the years 2009 and 2013, where we optimized the learning objective with instances from year INLINEFORM0 and evaluate on the years INLINEFORM1 (see Section SECREF46 )  The results show the precision–recall curve  The red curve shows baseline B1 BIBREF11 , and the blue one shows the performance of INLINEFORM2   The curve shows for varying confidence scores (high to low) the precision on labeling the pair INLINEFORM3 as `relevant'  In addition, at each confidence score we can compute the corresponding recall for the `relevant' label  For high confidence scores on labeling the news-entity pairs, the baseline B1 achieves on average a precision score of P=0 50, while INLINEFORM4 has P=0 93  We note that with the drop in the confidence score the corresponding precision and recall values drop too, and the overall F1 score for B1 is around F1=0 2, in contrast we achieve an average score of F1=0 67 
It is evident from Figure FIGREF55 that for the years 2009 and 2013, INLINEFORM0 significantly outperforms the baseline B1  We measure the significance through the t-test statistic and get a p-value of INLINEFORM1   The improvement we achieve over B1 in absolute numbers, INLINEFORM2 P=+0 5 in terms of precision for the years between 2009 and 2014, and a similar improvement in terms of F1 score  The improvement for recall is INLINEFORM3 R=+0 4  The relative improvement over B1 for P and F1 is almost 1 8 times better, while for recall we are 3 5 times better  In Table TABREF58 we show the overall scores for the evaluation metrics for B1 and INLINEFORM4 

ground_truth: irony accuracy, sentiment preservation

 irony accuracy and sentiment preservation

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What baseline model is used?
Context 2:  Finally, for B2 we achieve much poorer performance, with average scores of P=0 21, R=0 20 and F1=0 21 
Robustness  In Table TABREF58 , we show the overall performance for the years between 2009 and 2013  An interesting observation we make is that we have a very robust performance and the results are stable across the years  If we consider the experimental setup, where for year INLINEFORM0 we optimize the learning objective with only 74k training instances and evaluate on the rest of the instances, it achieves a very good performance  We predict with F1=0 68 the remaining 469k instances for the years INLINEFORM1  
The results are particularly promising considering the fact that the distribution between our two classes is highly skewed  On average the number of `relevant' pairs account for only around INLINEFORM0 of all pairs  A good indicator to support such a statement is the kappa (denoted by INLINEFORM1 ) statistic  INLINEFORM2 measures agreement between the algorithm and the gold standard on both labels while correcting for chance agreement (often expected due to extreme distributions)  The INLINEFORM3 scores for B1 across the years is on average INLINEFORM4 , while for INLINEFORM5 we achieve a score of INLINEFORM6 (the maximum score for INLINEFORM7 is 1) 
In Figure FIGREF60 we show the impact of the individual feature groups that contribute to the superior performance in comparison to the baselines  Relative entity frequency from the salience feature, models the entity salience as an exponentially decaying function based on the positional index of the paragraph where the entity appears  The performance of INLINEFORM0 with relative entity frequency from the salience feature group is close to that of all the features combined  The authority and novelty features account to a further improvement in terms of precision, by adding roughly a 7%-10% increase  However, if both feature groups are considered separately, they significantly outperform the baseline B1 
Article–Section Placement
We model the ASP placement task as a successor of the AEP task  For all the `relevant' news entity pairs, the task is to determine the correct entity section  Each section in a Wikipedia entity page represents a different topic  For example, Barack Obama has the sections `Early Life', `Presidency', `Family and Personal Life' etc  However, many entity pages have an incomplete section structure  Incomplete or missing sections are due to two Wikipedia properties  First, long-tail entities miss information and sections due to their lack of popularity  Second, for all entities whether popular or not, certain sections might occur for the first time due to real world developments  As an example, the entity Germanwings did not have an `Accidents' section before this year's disaster, which was the first in the history of the airline 
Even if sections are missing for certain entities, similar sections usually occur in other entities of the same class (e g  other airlines had disasters and therefore their pages have an accidents section)  We exploit such homogeneity of section structure and construct templates that we use to expand entity profiles  The learning objective for INLINEFORM0 takes into account the following properties:
Section-templates: account for incomplete section structure for an entity profile INLINEFORM0 by constructing section templates INLINEFORM1 from an entity class INLINEFORM2
Overall fit: measures the overall fit of a news article to sections in the section templates INLINEFORM0
Given the fact that entity profiles are often incomplete, we construct section templates for every entity class  We group entities based on their class INLINEFORM0 and construct section templates INLINEFORM1   For different entity classes, e g  Person and Location, the section structure and the information represented in those section varies heavily  Therefore, the section templates are with respect to the individual classes in our experimental setup (see Figure FIGREF42 )  DISPLAYFORM0
Generating section templates has two main advantages  Firstly, by considering class-based profiles, we can overcome the problem of incomplete individual entity profiles and thereby are able to suggest news articles to sections that do not yet exist in a specific entity INLINEFORM0   The second advantage is that we are able to canonicalize the sections, i e  `Early Life' and `Early Life and Childhood' would be treated similarly 
To generate the section template INLINEFORM0 , we extract all sections from entities of a given type INLINEFORM1 at year INLINEFORM2   Next, we cluster the entity sections, based on an extended version of k–means clustering BIBREF18 , namely x–means clustering introduced in Pelleg et al  which estimates the number of clusters efficiently BIBREF19   As a similarity metric we use the cosine similarity computed based on the tf–idf models of the sections  Using the x–means algorithm we overcome the requirement to provide the number of clusters k beforehand  x–means extends the k–means algorithm, such that a user only specifies a range [ INLINEFORM3 , INLINEFORM4 ] that the number of clusters may reasonably lie in 
The learning objective of INLINEFORM0 is to determine the overall fit of a news article INLINEFORM1 to one of the sections in a given section template INLINEFORM2   The template is pre-determined by the class of the entity for which the news is suggested as relevant by INLINEFORM3   In all cases, we measure how well INLINEFORM4 fits each of the sections INLINEFORM5 as well as the specific entity section INLINEFORM6   The section profiles in INLINEFORM7 represent the aggregated entity profiles from all entities of class INLINEFORM8 at year INLINEFORM9  
To learn INLINEFORM0 we rely on a variety of features that consider several similarity aspects as shown in Table TABREF31   For the sake of simplicity we do not make the distinction in Table TABREF31 between the individual entity section and class-based section similarities, INLINEFORM1 and INLINEFORM2 , respectively  Bear in mind that an entity section INLINEFORM3 might be present at year INLINEFORM4 but not at year INLINEFORM5 (see for more details the discussion on entity profile expansion in Section UID69 ) 
Topic  We use topic similarities to ensure (i) that the content of INLINEFORM0 fits topic-wise with a specific section text and (ii) that it has a similar topic to previously referred news articles in that section  In a pre-processing stage we compute the topic models for the news articles, entity sections INLINEFORM1 and the aggregated class-based sections in INLINEFORM2   The topic models are computed using LDA BIBREF20   We only computed a single topic per article/section as we are only interested in topic term overlaps between article and sections  We distinguish two main features: the first feature measures the overlap of topic terms between INLINEFORM3 and the entity section INLINEFORM4 and INLINEFORM5 , and the second feature measures the overlap of the topic model of INLINEFORM6 against referred news articles in INLINEFORM7 at time INLINEFORM8  
Syntactic  These features represent a mechanism for conveying the importance of a specific text snippet, solely based on the frequency of specific POS tags (i e  NNP, CD etc ), as commonly used in text summarization tasks  Following the same intuition as in BIBREF8 , we weigh the importance of articles by the count of specific POS tags  We expect that for different sections, the importance of POS tags will vary  We measure the similarity of POS tags in a news article against the section text  Additionally, we consider bi-gram and tri-gram POS tag overlap  This exploits similarity in syntactical patterns between the news and section text 
Lexical  As lexical features, we measure the similarity of INLINEFORM0 against the entity section text INLINEFORM1 and the aggregate section text INLINEFORM2   Further, we distinguish between the overall similarity of INLINEFORM3 and that of the different news paragraphs ( INLINEFORM4 which denotes the paragraphs of INLINEFORM5 up to the 5th paragraph)  A higher similarity on the first paragraphs represents a more confident indicator that INLINEFORM6 should be suggested to a specific section INLINEFORM7   We measure the similarity based on two metrics: (i) the KL-divergence between the computed language models and (ii) cosine similarity of the corresponding paragraph text INLINEFORM8 and section text 
Entity-based  Another feature set we consider is the overlap of named entities and their corresponding entity classes  For different entity sections, we expect to find a particular set of entity classes that will correlate with the section, e g  `Early Life' contains mostly entities related to family, school, universities etc 
Frequency  Finally, we gather statistics about the number of entities, paragraphs, news article length, top– INLINEFORM0 entities and entity classes, and the frequency of different POS tags  Here we try to capture patterns of articles that are usually cited in specific sections 
Evaluation Plan
In this section we outline the evaluation plan to verify the effectiveness of our learning approaches  To evaluate the news suggestion problem we are faced with two challenges 
What comprises the ground truth for such a task ?
How do we construct training and test splits given that entity pages consists of text added at different points in time ?
Consider the ground truth challenge  Evaluating if an arbitrary news article should be included in Wikipedia is both subjective and difficult for a human if she is not an expert  An invasive approach, which was proposed by Barzilay and Sauper BIBREF8 , adds content directly to Wikipedia and expects the editors or other users to redact irrelevant content over a period of time  The limitations of such an evaluation technique is that content added to long-tail entities might not be evaluated by informed users or editors in the experiment time frame  It is hard to estimate how much time the added content should be left on the entity page  A more non-invasive approach could involve crowdsourcing of entity and news article pairs in an IR style relevance assessment setup  The problem of such an approach is again finding knowledgeable users or experts for long-tail entities  Thus the notion of relevance of a news recommendation is challenging to evaluate in a crowd setup 
We take a slightly different approach by making an assumption that the news articles already present in Wikipedia entity pages are relevant  To this extent, we extract a dataset comprising of all news articles referenced in entity pages (details in Section SECREF40 )  At the expense of not evaluating the space comprising of news articles absent in Wikipedia, we succeed in (i) avoiding restrictive assumptions about the quality of human judgments, (ii) being invasive and polluting Wikipedia, and (iii) deriving a reusable test bed for quicker experimentation 
The second challenge of construction of training and test set separation is slightly easier and is addressed in Section SECREF46  
Datasets
The datasets we use for our experimental evaluation are directly extracted from the Wikipedia entity pages and their revision history  The generated data represents one of the contributions of our paper  The datasets are the following:
Entity Classes  We focus on a manually predetermined set of entity classes for which we expect to have news coverage  The number of analyzed entity classes is 27, including INLINEFORM0 entities with at least one news reference  The entity classes were selected from the DBpedia class ontology  Figure FIGREF42 shows the number of entities per class for the years (2009-2014) 
News Articles  We extract all news references from the collected Wikipedia entity pages  The extracted news references are associated with the sections in which they appear  In total there were INLINEFORM0 news references, and after crawling we end up with INLINEFORM1 successfully crawled news articles  The details of the news article distribution, and the number of entities and sections from which they are referred are shown in Table TABREF44  
Article-Entity Ground-truth  The dataset comprises of the news and entity pairs INLINEFORM0   News-entity pairs are relevant if the news article is referenced in the entity page  Non-relevant pairs (i e  negative training examples) consist of news articles that contain an entity but are not referenced in that entity's page  If a news article INLINEFORM1 is referred from INLINEFORM2 at year INLINEFORM3 , the features are computed taking into account the entity profiles at year INLINEFORM4  
Article-Section Ground-truth  The dataset consists of the triple INLINEFORM0 , where INLINEFORM1 , where we assume that INLINEFORM2 has already been determined as relevant  We therefore have a multi-class classification problem where we need to determine the section of INLINEFORM3 where INLINEFORM4 is cited  Similar to the article-entity ground truth, here too the features compute the similarity between INLINEFORM5 , INLINEFORM6 and INLINEFORM7  
Data Pre-Processing
We POS-tag the news articles and entity profiles INLINEFORM0 with the Stanford tagger BIBREF21   For entity linking the news articles, we use TagMe! BIBREF22 with a confidence score of 0 3  On a manual inspection of a random sample of 1000 disambiguated entities, the accuracy is above 0 9  On average, the number of entities per news article is approximately 30  For entity linking the entity profiles, we simply follow the anchor text that refers to Wikipedia entities 
Train and Testing Evaluation Setup
We evaluate the generated supervised models for the two tasks, AEP and ASP, by splitting the train and testing instances  It is important to note that for the pairs INLINEFORM0 and the triple INLINEFORM1 , the news article INLINEFORM2 is referenced at time INLINEFORM3 by entity INLINEFORM4 , while the features take into account the entity profile at time INLINEFORM5   This avoids any `overlapping' content between the news article and the entity page, which could affect the learning task of the functions INLINEFORM6 and INLINEFORM7   Table TABREF47 shows the statistics of train and test instances  We learn the functions at year INLINEFORM8 and test on instances for the years greater than INLINEFORM9   Please note that we do not show the performance for year 2014 as we do not have data for 2015 for evaluation 
Article-Section Placement
Here we show the evaluation setup for ASP task and discuss the results with a focus on three main aspects, (i) the overall performance across the years, (ii) the entity class specific performance, and (iii) the impact on entity profile expansion by suggesting missing sections to entities based on the pre-computed templates 
Baselines  To the best of our knowledge, we are not aware of any comparable approach for this task  Therefore, the baselines we consider are the following:
S1: Pick the section from template INLINEFORM0 with the highest lexical similarity to INLINEFORM1 : S1 INLINEFORM2
S2: Place the news into the most frequent section in INLINEFORM0
Learning Models  We use Random Forests (RF) BIBREF23 and Support Vector Machines (SVM) BIBREF24   The models are optimized taking into account the features in Table TABREF31   In contrast to the AEP task, here the scale of the number of instances allows us to learn the SVM models  The SVM model is optimized using the INLINEFORM0 loss function and uses the Gaussian kernels 
Metrics  We compute precision P as the ratio of news for which we pick a section INLINEFORM0 from INLINEFORM1 and INLINEFORM2 conforms to the one in our ground-truth (see Section SECREF40 )  The definition of recall R and F1 score follows from that of precision 
Figure FIGREF66 shows the overall performance and a comparison of our approach (when INLINEFORM0 is optimized using SVM) against the best performing baseline S2  With the increase in the number of training instances for the ASP task the performance is a monotonically non-decreasing function  For the year 2009, we optimize the learning objective of INLINEFORM1 with around 8% of the total instances, and evaluate on the rest  The performance on average is around P=0 66 across all classes  Even though for many classes the performance is already stable (as we will see in the next section), for some classes we improve further  If we take into account the years between 2010 and 2012, we have an increase of INLINEFORM2 P=0 17, with around 70% of instances used for training and the remainder for evaluation  For the remaining years the total improvement is INLINEFORM3 P=0 18 in contrast to the performance at year 2009 
On the other hand, the baseline S1 has an average precision of P=0 12  The performance across the years varies slightly, with the year 2011 having the highest average precision of P=0 13  Always picking the most frequent section as in S2, as shown in Figure FIGREF66 , results in an average precision of P=0 17, with a uniform distribution across the years 
Here we show the performance of INLINEFORM0 decomposed for the different entity classes  Specifically we analyze the 27 classes in Figure FIGREF42   In Table TABREF68 , we show the results for a range of years (we omit showing all years due to space constraints)  For illustration purposes only, we group them into four main classes ( INLINEFORM1 Person, Organization, Location, Event INLINEFORM2 ) and into the specific sub-classes shown in the second column in Table TABREF68   For instance, the entity classes OfficeHolder and Politician are aggregated into Person–Politics 
It is evident that in the first year the performance is lower in contrast to the later years  This is due to the fact that as we proceed, we can better generalize and accurately determine the correct fit of an article INLINEFORM0 into one of the sections from the pre-computed templates INLINEFORM1   The results are already stable for the year range INLINEFORM2   For a few Person sub-classes, e g  Politics, Entertainment, we achieve an F1 score above 0 9  These additionally represent classes with a sufficient number of training instances for the years INLINEFORM3   The lowest F1 score is for the Criminal and Television classes  However, this is directly correlated with the insufficient number of instances 
The baseline approaches for the ASP task perform poorly  S1, based on lexical similarity, has a varying performance for different entity classes  The best performance is achieved for the class Person – Politics, with P=0 43  This highlights the importance of our feature choice and that the ASP cannot be considered as a linear function, where the maximum similarity yields the best results  For different entity classes different features and combination of features is necessary  Considering that S2 is the overall best performing baseline, through our approach INLINEFORM0 we have a significant improvement of over INLINEFORM1 P=+0 64 
The models we learn are very robust and obtain high accuracy, fulfilling our pre-condition for accurate news suggestions into the entity sections  We measure the robustness of INLINEFORM0 through the INLINEFORM1 statistic  In this case, we have a model with roughly 10 labels (corresponding to the number of sections in a template INLINEFORM2 )  The score we achieve shows that our model predicts with high confidence with INLINEFORM3  
The last analysis is the impact we have on expanding entity profiles INLINEFORM0 with new sections  Figure FIGREF70 shows the ratio of sections for which we correctly suggest an article INLINEFORM1 to the right section in the section template INLINEFORM2   The ratio here corresponds to sections that are not present in the entity profile at year INLINEFORM3 , that is INLINEFORM4   However, given the generated templates INLINEFORM5 , we can expand the entity profile INLINEFORM6 with a new section at time INLINEFORM7   In details, in the absence of a section at time INLINEFORM8 , our model trains well on similar sections from the section template INLINEFORM9 , hence we can predict accurately the section and in this case suggest its addition to the entity profile  With time, it is obvious that the expansion rate decreases at later years as the entity profiles become more `complete' 
This is particularly interesting for expanding the entity profiles of long-tail entities as well as updating entities with real-world emerging events that are added constantly  In many cases such missing sections are present at one of the entities of the respective entity class INLINEFORM0   An obvious case is the example taken in Section SECREF16 , where the `Accidents' is rather common for entities of type Airline  However, it is non-existent for some specific entity instances, i e Germanwings airline 
Through our ASP approach INLINEFORM0 , we are able to expand both long-tail and trunk entities  We distinguish between the two types of entities by simply measuring their section text length  The real distribution in the ground truth (see Section SECREF40 ) is 27% and 73% are long-tail and trunk entities, respectively  We are able to expand the entity profiles for both cases and all entity classes without a significant difference, with the only exception being the class Creative Work, where we expand significantly more trunk entities 
Conclusion and Future Work
In this work, we have proposed an automated approach for the novel task of suggesting news articles to Wikipedia entity pages to facilitate Wikipedia updating  The process consists of two stages  In the first stage, article–entity placement, we suggest news articles to entity pages by considering three main factors, such as entity salience in a news article, relative authority and novelty of news articles for an entity page  In the second stage, article–section placement, we determine the best fitting section in an entity page  Here, we remedy the problem of incomplete entity section profiles by constructing section templates for specific entity classes  This allows us to add missing sections to entity pages  We carry out an extensive experimental evaluation on 351,983 news articles and 73,734 entities coming from 27 distinct entity classes  For the first stage, we achieve an overall performance with P=0 93, R=0 514 and F1=0 676, outperforming our baseline competitors significantly  For the second stage, we show that we can learn incrementally to determine the correct section for a news article based on section templates  The overall performance across different classes is P=0 844, R=0 885 and F1=0 860 
In the future, we will enhance our work by extracting facts from the suggested news articles  Results suggest that the news content cited in entity pages comes from the first paragraphs  However, challenging task such as the canonicalization and chronological ordering of facts, still remain 
Context 3: Introduction
BioASQ is a biomedical document classification, document retrieval, and question answering competition, currently in its seventh year  We provide an overview of our submissions to semantic question answering task (7b, Phase B) of BioASQ 7 (except for 'ideal answer' test, in which we did not participate this year)  In this task systems are provided with biomedical questions and are required to submit ideal and exact answers to those questions  We have used BioBERT BIBREF0 based system , see also Bidirectional Encoder Representations from Transformers(BERT) BIBREF1, and we fine tuned it for the biomedical question answering task  Our system scored near the top for factoid questions for all the batches of the challenge  More specifially, in the third test batch set, our system achieved highest ‘MRR’ score for Factoid Question Answering task  Also, for List-type question answering task our system achieved highest recall score in the fourth test batch set  Along with our detailed approach, we present the results for our submissions and also highlight identified downsides for our current approach and ways to improve them in our future experiments  In last test batch results we placed 4th for List-type questions and 3rd for Factoid-type questions )
The QA task is organized in two phases  Phase A deals with retrieval of the relevant document, snippets, concepts, and RDF triples, and phase B deals with exact and ideal answer generations (which is a paragraph size summary of snippets)  Exact answer generation is required for factoid, list, and yes/no type question 
BioASQ organizers provide the training and testing data  The training data consists of questions, gold standard documents, snippets, concepts, and ideal answers (which we did not use in this paper, but we used last year BIBREF2)  The test data is split between phases A and B  The Phase A dataset consists of the questions, unique ids, question types  The Phase B dataset consists of the questions, golden standard documents, snippets, unique ids and question types  Exact answers for factoid type questions are evaluated using strict accuracy (the top answer), lenient accuracy (the top 5 answers), and MRR (Mean Reciprocal Rank) which takes into account the ranks of returned answers  Answers for the list type question are evaluated using precision, recall, and F-measure 
Related Work ::: BioAsq
Sharma et al  BIBREF3 describe a system with two stage process for factoid and list type question answering  Their system extracts relevant entities and then runs supervised classifier to rank the entities  Wiese et al  BIBREF4 propose neural network based model for Factoid and List-type question answering task  The model is based on Fast QA and predicts the answer span in the passage for a given question  The model is trained on SQuAD data set and fine tuned on the BioASQ data  Dimitriadis et al  BIBREF5 proposed two stage process for Factoid question answering task  Their system uses general purpose tools such as Metamap, BeCas to identify candidate sentences  These candidate sentences are represented in the form of features, and are then ranked by the binary classifier  Classifier is trained on candidate sentences extracted from relevant questions, snippets and correct answers from BioASQ challenge  For factoid question answering task highest ‘MRR’ achieved in the 6th edition of BioASQ competition is ‘0 4325’  Our system is a neural network model based on contextual word embeddings BIBREF1 and achieved a ‘MRR’ score ‘0 6103’ in one of the test batches for Factoid Question Answering task 
Related Work ::: A minimum background on BERT
BERT stands for "Bidirectional Encoder Representations from Transformers" BIBREF1 is a contextual word embedding model  Given a sentence as an input, contextual embedding for the words are returned  The BERT model was designed so it can be fine tuned for 11 different tasks BIBREF1, including question answering tasks  For a question answering task, question and paragraph (context) are given as an input  A BERT standard is that question text and paragraph text are separated by a separator [Sep]  BERT question-answering fine tuning involves adding softmax layer  Softmax layer takes contextual word embeddings from BERT as input and learns to identity answer span present in the paragraph (context)  This process is represented in Figure FIGREF4 
BERT was originally trained to perform tasks such as language model creation using masked words and next-sentence-prediction  In other words BERT weights are learned such that context is used in building the representation of the word, not just as a loss function to help learn a context-independent representation  For detailed understanding of BERT Architecture, please refer to the original BERT paper BIBREF1 
Related Work ::: A minimum background on BERT ::: Comparison of Word Embeddings and Contextual Word Embeddings
A ‘word embedding’ is a learned representation  It is represented in the form of vector where words that have the same meaning have a similar vector representation  Consider a word embedding model 'word2vec' BIBREF6 trained on a corpus  Word embeddings generated from the model are context independent that is, word embeddings are returned regardless of where the words appear in a sentence and regardless of e g  the sentiment of the sentence  However, contextual word embedding models like BERT also takes context of the word into consideration 
Related Work ::: Comparison of BERT and Bio-BERT
‘BERT’ and BioBERT are very similar in terms of architecture  Difference is that ‘BERT’ is pretrained on Wikipedia articles, whereas BioBERT version used in our experiments is pretrained on Wikipedia, PMC and PubMed articles  Therefore BioBERT model is expected to perform well with biomedical text, in terms of generating contextual word embeddings 
BioBERT model used in our experiments is based on BERT-Base Architecture; BERT-Base has 12 transformer Layers where as BERT-Large has 24 transformer layers  Moreover contextual word embedding vector size is 768 for BERT-Base and more for BERT-large  According to BIBREF1 Bert-Large, fine-tuned on SQuAD 1 1 question answering data BIBREF7 can achieve F1 Score of 90 9 for Question Answering task where as BERT-Base Fine-tuned on the same SQuAD question answering BIBREF7 data could achieve F1 score of 88 5  One downside of the current version BioBERT is that word-piece vocabulary is the same as that of original BERT Model, as a result word-piece vocabulary does not include biomedical jargon  Lee et al  BIBREF0 created BioBERT, using the same pre-trained BERT released by Google, and hence in the word-piece vocabulary (vocab txt), as a result biomedical jargon is not included in word-piece vocabulary  Modifying word-piece vocabulary (vocab txt) at this stage would loose original compatibility with ‘BERT’, hence it is left unmodified 
In our future work we would like to build pre-trained ‘BERT’ model from scratch  We would pretrain the model with biomedical corpus (PubMed, ‘PMC’) and Wikipedia  Doing so would give us scope to create word piece vocabulary to include biomedical jargon and there are chances of model performing better with biomedical jargon being included in the word piece vocabulary  We will consider this scenario in the future, or wait for the next version of BioBERT 
Experiments: Factoid Question Answering Task
For Factoid Question Answering task, we fine tuned BioBERT BIBREF0 with question answering data and added new features  Fig  FIGREF4 shows the architecture of BioBERT fine tuned for question answering tasks: Input to BioBERT is word tokenized embeddings for question and the paragraph (Context)  As per the ‘BERT’ BIBREF1 standards, tokens ‘[CLS]’ and ‘[SEP]’ are appended to the tokenized input as illustrated in the figure  The resulting model has a softmax layer formed for predicting answer span indices in the given paragraph (Context)  On test data, the fine tuned model generates $n$-best predictions for each question  For a question, $n$-best corresponds that $n$ answers are returned as possible answers in the decreasing order of confidence  Variable $n$ is configurable  In our paper, any further mentions of ‘answer returned by the model’ correspond to the top answer returned by the model 
Experiments: Factoid Question Answering Task ::: Setup
BioASQ provides the training data  This data is based on previous BioASQ competitions  Train data we have considered is aggregate of all train data sets till the 5th version of BioASQ competition  We cleaned the data, that is, question-answering data without answers are removed and left with a total count of ‘530’ question answers  The data is split into train and test data in the ratio of 94 to 6; that is, count of '495' for training and '35' for testing 
The original data format is converted to the BERT/BioBERT format, where BioBERT expects ‘start_index’ of the actual answer  The ‘start_index corresponds to the index of the answer text present in the paragraph/ Context  For finding ‘start_index’ we used built-in python function find()  The function returns the lowest index of the actual answer present in the context(paragraph)  If the answer is not found ‘-1’ is returned as the index  The efficient way of finding start_index is, if the paragraph (Context) has multiple instances of answer text, then ‘start_index’ of the answer should be that instance of answer text whose context actually matches with what’s been asked in the question 
Example (Question, Answer and Paragraph from BIBREF8):
Question: Which drug should be used as an antidote in benzodiazepine overdose?
Answer: 'Flumazenil'
Paragraph(context):
"Flumazenil use in benzodiazepine overdose in the UK: a retrospective survey of NPIS data  OBJECTIVE: Benzodiazepine (BZD) overdose (OD) continues to cause significant morbidity and mortality in the UK  Flumazenil is an effective antidote but there is a risk of seizures, particularly in those who have co-ingested tricyclic antidepressants  A study was undertaken to examine the frequency of use, safety and efficacy of flumazenil in the management of BZD OD in the UK  METHODS: A 2-year retrospective cohort study was performed of all enquiries to the UK National Poisons Information Service involving BZD OD  RESULTS: Flumazenil was administered to 80 patients in 4504 BZD-related enquiries, 68 of whom did not have ventilatory failure or had recognised contraindications to flumazenil  Factors associated with flumazenil use were increased age, severe poisoning and ventilatory failure  Co-ingestion of tricyclic antidepressants and chronic obstructive pulmonary disease did not influence flumazenil administration  Seizure frequency in patients not treated with flumazenil was 0 3%" 
Actual answer is 'Flumazenil', but there are multiple instances of word 'Flu-mazenil'  Efficient way to identify the start-index for 'Flumazenil'(answer) is to find that particular instance of the word 'Flumazenil' which matches the context for the question  In the above example 'Flumazenil' highlighted in bold is the actual instance that matches question's context  Unfortunately, we could not identify readily available tools that can achieve this goal  In our future work, we look forward to handling these scenarios effectively 
Note: The creators of 'SQuAD' BIBREF7 have handled the task of identifying answer's start_index effectively  But 'SQuAD' data set is much more general and does not include biomedical question answering data 
Experiments: Factoid Question Answering Task ::: Training and error analysis
During our training with the BioASQ data, learning rate is set to 3e-5, as mentioned in the BioBERT paper BIBREF0  We started training the model with 495 available train data and 35 test data by setting number of epochs to 50  After training with these hyper-parameters training accuracy(exact match) was 99 3%(overfitting) and testing accuracy is only 4%  In the next iteration we reduced the number of epochs to 25 then training accuracy is reduced to 98 5% and test accuracy moved to 5%  We further reduced number of epochs to 15, and the resulting training accuracy was 70% and test accuracy 15%  In the next iteration set number of epochs to 12 and achieved train accuracy of 57 7% and test accuracy 23 3%  Repeated the experiment with 11 epochs and found training accuracy to be 57 7% and test accuracy to be same 22%  In the next iteration we set number of epochs to '9' and found training accuracy of 48% and test accuracy of 15%  Hence optimum number of epochs is taken as 12 epochs 
During our error analysis we found that on test data, model tends to return text in the beginning of the context(paragraph) as the answer  On analysing train data, we found that there are '120'(out of '495') question answering data instances having start_index:0, meaning 120( 25%) question answering data has first word(s) in the context(paragraph) as the answer  We removed 70% of those instances in order to make train data more balanced  In the new train data set we are left with '411' question answering data instances  This time we got the highest test accuracy of 26% at 11 epochs  We have submitted our results for BioASQ test batch-2, got strict accuracy of 32% and our system stood in 2nd place  Initially, hyper-parameter- 'batch size' is set to ‘400’  Later it is tuned to '32'  Although accuracy(exact answer match) remained at 26%, model generated concise and better answers at batch size ‘32’, that is wrong answers are close to the expected answer in good number of cases 
Example (from BIBREF8)
Question: Which mutated gene causes Chediak Higashi Syndrome?
Exact Answer: ‘lysosomal trafficking regulator gene’ 
The answer returned by a model trained at ‘400’ batch size is ‘Autosomal-recessive complicated spastic paraplegia with a novel lysosomal trafficking regulator’, and from the one trained at ‘32’ batch size is ‘lysosomal trafficking regulator’ 
In further experiments, we have fine tuned the BioBERT model with both ‘SQuAD’ dataset (version 2 0) and BioAsq train data  For training on ‘SQuAD’, hyper parameters- Learning rate and number of epochs are set to ‘3e-3’ and ‘3’ respectively as mentioned in the paper BIBREF1  Test accuracy of the model boosted to 44%  In one more experiment we trained model only on ‘SQuAD’ dataset, this time test accuracy of the model moved to 47%  The reason model did not perform up to the mark when trained with ‘SQuAD’ alongside BioASQ data could be that in formatted BioASQ data, start_index for the answer is not accurate, and affected the overall accuracy 
Our Systems and Their Performance on Factoid Questions
We have experimented with several systems and their variations, e g  created by training with specific additional features (see next subsection)  Here is their list and short descriptions  Unfortunately we did not pay attention to naming, and the systems evolved between test batches, so the overall picture can only be understood by looking at the details 
When we started the experiments our objective was to see whether BioBERT and entailment-based techniques can provide value for in the context of biomedical question answering  The answer to both questions was a yes, qualified by many examples clearly showing the limitations of both methods  Therefore we tried to address some of these limitations using feature engineering with mixed results: some clear errors got corrected and new errors got introduced, without overall improvement but convincing us that in future experiments it might be worth trying feature engineering again especially if more training data were available 
Overall we experimented with several approaches with the following aspects of the systems changing between batches, that is being absent or present:
training on BioAsq data vs  training on SQuAD
using the BioAsq snippets for context vs  using the documents from the provided URLs for context
adding or not the LAT, i e  lexical answer type, feature (see BIBREF9, BIBREF10 and an explanation in the subsection just below) 
For Yes/No questions (only) we experimented with the entailment methods 
We will discuss the performance of these models below and in Section 6  But before we do that, let us discuss a feature engineering experiment which eventually produced mixed results, but where we feel it is potentially useful in future experiments 
Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative)
During error analysis we found that for some cases, answer being returned by the model is far away from what it is being asked in the Question 
Example: (from BIBREF8)
Question: Hy's law measures failure of which organ?
Actual Answer: ‘Liver’ 
The answer returned by one of our models was ‘alanine aminotransferase’, which is an enzyme  The model returns an enzyme, when the question asked for the organ name  To address this type of errors, we decided to try the concepts of ‘Lexical Answer Type’ (LAT) and Focus Word, which was used in IBM Watson, see BIBREF11 for overview; BIBREF10 for technical details, and BIBREF9 for details on question analysis  In an example given in the last source we read:
POETS & POETRY: He was a bank clerk in the Yukon before he published "Songs of a Sourdough" in 1907 
The focus is the part of the question that is a reference to the answer  In the example above, the focus is "he" 
LATs are terms in the question that indicate what type of entity is being asked for 
The headword of the focus is generally a LAT, but questions often contain additional LATs, and in the Jeopardy! domain, categories are an additional source of LATs 
(   ) In the example, LATs are "he", "clerk", and "poet" 
For example in the question "Which plant does oleuropein originate from?" (BIBREF8)  The LAT here is ‘plant’  For the BioAsq task we did not need to explicitly distinguish between the focus and the LAT concepts  In this example, the expectation is that answer returned by the model is a plant  Thus it is conceivable that the cosine distance between contextual embedding of word 'plant' in the question and contextual embedding for the answer present in the paragraph(context) is comparatively low  As a result model learns to adjust its weights during training phase and returns answers with low cosine distance with the LAT 
We used Stanford CoreNLP BIBREF12 library to write rules for extracting lexical answer type present in the question, both 'parts of speech'(POS) and dependency parsing functionality was used  We incorporated the Lexical Answer Type into one of our systems, UNCC_QA1 in Batch 4  This system underperformed our system FACTOIDS by about 3% in the MRR measure, but corrected errors such as in the example above 
Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative) ::: Assumptions and rules for deriving lexical answer type 
There are different question types: ‘Which’, ‘What’, ‘When’, ‘How’ etc  Each type of question is being handled differently and there are commonalities among the rules written for different question types  Question words are identified through parts of speech tags: 'WDT', 'WRB' ,'WP'  We assumed that LAT is a ‘Noun’ and follows the question word  Often it was also a subject (nsubj)  This process is illustrated in Fig FIGREF15 
LAT computation was governed by a few simple rules, e g  when a question has multiple words that are 'Subjects’ (and ‘Noun’), a word that is in proximity to the question word is considered as ‘LAT’  These rules are different for each "Wh" word 
Namely, when the word immediately following the question word is a Noun, window size is set to ‘3’  The window size ‘3’ means we iterate through the next ‘3’ words to check if any of the word is both Noun and Subject, If so, such word is considered the ‘LAT’; else the word that is present very next to the question word is considered as the ‘LAT’ 
For questions with words ‘Which’ , ‘What’, ‘When’; a Noun immediately following the question word is very often the LAT, e g  'enzyme' in Which enzyme is targeted by Evolocumab?  When the word immediately following the question word is not a Noun, e g  in What is the function of the protein Magt1? the window size is set to ‘5’, and we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject  If present, the word is considered as the ‘LAT’; else, the Noun in close proximity to the question word and following it is returned as the ‘LAT’ 
For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself  For the word ‘How', e g  in How many selenoproteins are encoded in the human genome?, we look at the adjective and if we find one, we take it to be the LAT, otherwise the word 'How' is considered as the ‘LAT’ 
Perhaps because of using only very simple rules, the accuracy for ‘LAT’ derivation is 75%; that is, in the remaining 25% of the cases the LAT word is identified incorrectly  Worth noting is that the overall performance the system that used LATs was slightly inferior to the system without LATs, but the types of errors changed  The training used BioBERT with the LAT feature as part of the input string  The errors it introduces usually involve finding the wrong element of the correct type e g  wrong enzyme when two similar enzymes are described in the text, or 'neuron' when asked about a type of cell with a certain function, when the answer calls for a different cell category, adipocytes, and both are mentioned in the text  We feel with more data and additional tuning or perhaps using an ensemble model, we might be able to keep the correct answers, and improve the results on the confusing examples like the one mentioned above  Therefore if we improve our ‘LAT’ derivation logic, or have larger datasets, then perhaps the neural network techniques they will yield better results 
Our Systems and Their Performance on Factoid Questions ::: Impact of Training using BioAsq data (slightly negative)
Training on BioAsq data in our entry in Batch 1 and Batch 2 under the name QA1 showed it might lead to overfitting  This happened both with (Batch 2) and without (Batch 1) hyperparameters tuning: abysmal 18% MRR in Batch 1, and slighly better one, 40% in Batch 2 (although in Batch 2 it was overall the second best result in MRR but 16% lower than the highest score) 
In Batch 3 (only), our UNCC_QA3 system was fine tuned on BioAsq and SQuAD 2 0 BIBREF7, and for data preprocessing Context paragraph is generated from relevant snippets provided in the test data  This system underperformed, by about 2% in MRR, our other entry UNCC_QA1, which was also an overall category winner for this batch  The latter was also trained on SQuAD, but not on BioAsq  We suspect that the reason could be the simplistic nature of the find() function described in Section 3 1  So, this could be an area where a better algorithm for finding the best occurrence of an entity could improve performance 
Our Systems and Their Performance on Factoid Questions ::: Impact of Using Context from URLs (negative)
In some experiments, for context in testing, we used documents for which URL pointers are provided in BioAsq  However, our system UNCC_QA3 underperformed our other system tested only on the provided snippets 
In Batch 5 the underperformance was about 6% of MRR, compared to our best system UNCC_QA1, and by 9% to the top performer 
Performance on Yes/No and List questions
Our work focused on Factoid questions  But we also have done experiments on List-type and Yes/No questions 
Performance on Yes/No and List questions ::: Entailment improves Yes/No accuracy
We started by answering always YES (in batch 2 and 3) to get the baseline performance  For batch 4 we used entailment  Our algorithm was very simple: Given a question we iterate through the candidate sentences and try to find any candidate sentence is contradicting the question (with confidence over 50%), if so 'No' is returned as answer, else 'Yes' is returned  In batch 4 this strategy produced better than the BioAsq baseline performance, and compared to our other systems, the use of entailment increased the performance by about 13% (macro F1 score)  We used 'AllenNlp' BIBREF13 entailment library to find entailment of the candidate sentences with question 
Performance on Yes/No and List questions ::: For List-type the URLs have negative impact
Overall, we followed the similar strategy that's been followed for Factoid Question Answering task  We started our experiment with batch 2, where we submitted 20 best answers (with context from snippets)  Starting with batch 3, we performed post processing: once models generate answer predictions (n-best predictions), we do post-processing on the predicted answers  In test batch 4, our system (called FACTOIDS) achieved highest recall score of ‘0 7033’ but low precision of 0 1119, leaving open the question of how could we have better balanced the two measures 
In the post-processing phase, we take the top ‘20’ (batch 3) and top 5 (batch 4 and 5), predicted answers, tokenize them using common separators: 'comma' , 'and', 'also', 'as well as'  Tokens with characters count more than ‘100’ are eliminated and rest of the tokens are added to the list of possible answers  BioASQ evaluation mechanism does not consider snippets with more than ‘100’ characters as a valid answer  Considering lengthy snippets in to the list of answers would reduce the mean precision score  As a final step, duplicate snippets in the answer pool are removed  For example, consider these top 3 answers predicted by the system (before post-processing):
{
"text": "dendritic cells",
"probability": 0 7554540733426441,
"start_logit": 8 466046333312988,
"end_logit": 9 536355018615723
},
{
"text": "neutrophils, macrophages and
distinct subtypes of dendritic cells",
"probability": 0 13806867348304214,
"start_logit": 6 766478538513184,
"end_logit": 9 536355018615723
},
{
"text": "macrophages and distinct subtypes of dendritic",
"probability": 0 013973475271178242,
"start_logit": 6 766478538513184,
"end_logit": 7 24576473236084
},
After execution of post-processing heuristics, the list of answers returned is as follows:
["dendritic cells"],
["neutrophils"],
["macrophages"],
["distinct subtypes of dendritic cells"]
Summary of our results
The tables below summarize all our results  They show that the performance of our systems was mixed  The simple architectures and algorithm we used worked very well only in Batch 3  However, we feel we can built a better system based on this experience  In particular we observed both the value of contextual embeddings and of feature engineering (LAT), however we failed to combine them properly 
Summary of our results ::: Factoid questions ::: Systems used in Batch 5 experiments
System description for ‘UNCC_QA1’: The system was finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data 
System description for ‘QA1’ : ‘LAT’ feature was added and finetuned with SQuAD 2 0  For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data 
System description for ‘UNCC_QA3’ : Fine tuning process is same as it is done for the system ‘UNCC_QA1’ in test batch-5  Difference is during data preprocessing, Context/paragraph is generated from the relevant documents for which URLS are included in the test data 
Summary of our results ::: List Questions
For List-type questions, although post processing helped in the later batches, we never managed to obtain competitive precision, although our recall was good 
Summary of our results ::: Yes/No questions
The only thing worth remembering from our performance is that using entailment can have a measurable impact (at least with respect to a weak baseline)  The results (weak) are in Table 3 
Discussion, Future Experiments, and Conclusions ::: Summary:
In contrast to 2018, when we submitted BIBREF2 to BioASQ a system based on extractive summarization (and scored very high in the ideal answer category), this year we mainly targeted factoid question answering task and focused on experimenting with BioBERT  After these experiments we see the promise of BioBERT in QA tasks, but we also see its limitations  The latter we tried to address with mixed results using feature engineering  Overall these experiments allowed us to secure a best and a second best score in different test batches  Along with Factoid-type question, we also tried ‘Yes/No’ and ‘List’-type questions, and did reasonably well with our very simple approach 
For Yes/No the moral worth remembering is that reasoning has a potential to influence results, as evidenced by our adding the AllenNLP entailment BIBREF13 system increased its performance 
All our data and software is available at Github, in the previously referenced URL (end of Section 2) 
Discussion, Future Experiments, and Conclusions ::: Future experiments
In the current model, we have a shallow neural network with a softmax layer for predicting answer span  Shallow networks however are not good at generalizations  In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span  The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network  The mentioned dense layered question answering neural network need to be tuned for finding right hyper parameters  An example of such architecture is shown in Fig FIGREF30 
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network  By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy  Adding ‘LAT’ feature this way instead of feeding this word piece embedding directly to the BioBERT (as we did in our above experiments) would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'  Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy 
We also see potential for incorporating domain specific inference into the task e g  using the MedNLI dataset BIBREF14  For all types of experiments it might be worth exploring clinical BERT embeddings BIBREF15, explicitly incorporating domain knowledge (e g  BIBREF16) and possibly deeper discourse representations (e g  BIBREF17) 
APPENDIX
In this appendix we provide additional details about the implementations 
APPENDIX ::: Systems and their descriptions:
We used several variants of our systems when experimenting with the BioASQ problems  In retrospect, it would be much easier to understand the changes if we adopted some mnemonic conventions in naming the systems  So, we apologize for the names that do not reflect the modifications, and necessitate this list 
APPENDIX ::: Systems and their descriptions: ::: Factoid Type Question Answering:
We preprocessed the test data to convert test data to BioBERT format, We generated Context/paragraph by either aggregating relevant snippets provided or by aggregating documents for which URLS are provided in the BioASQ test data 
APPENDIX ::: Systems and their descriptions: ::: System description for QA1:
We generated Context/paragraph by aggregating relevant snippets available in the test data and mapped it against the question text and question id  We ignored the content present in the documents (document URLS were provided in the original test data)  The model is finetuned with BioASQ data 
data preprocessing is done in the same way as it is done for test batch-1  Model fine tuned on BioASQ data 
‘LAT’/ Focus word feature added and fine tuned with SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA_1:
System is finetuned on the SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
‘LAT’/ Focus word feature added and fine tuned with SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
The System is finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA3:
System is finetuned on the SQuAD 2 0 [reference] and BioASQ dataset[] For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
Fine tuning process is same as it is done for the system ‘UNCC_QA_1’ in test batch-5  Difference is during data preprocessing, Context/paragraph is generated form from the relevant documents for which URLS are included in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA2:
Fine tuning process is same as for ‘UNCC_QA_1 ’  Difference is Context/paragraph is generated form from the relevant documents for which URLS are included in the test data  System ‘UNCC_QA_1’ got the highest ‘MRR’ score in the 3rd test batch set 
APPENDIX ::: Systems and their descriptions: ::: System description for FACTOIDS:
The System is finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: List Type Questions:
We attempted List type questions starting from test batch ‘2’  Used similar approach that's been followed for Factoid Question answering task  For all the test batch sets, in the data pre processing phase Context/ paragraph is generated either by aggregating relevant snippets or by aggregating documents(URLS) provided in the BioASQ test data 
For test batch-2, model (System: QA1) is finetuned on BioASQ data and submitted top ‘20’ answers predicted by the model as the list of answers  system ‘QA1’ achieved low F-Measure score:‘0 0786’ in the second test batch  In the further test batches for List type questions, we finetuned the model on Squad data set [reference], implemented post processing techniques (refer section 5 2) and achieved a better F-measure score: ‘0 2862’ in the final test batch set 
In test batch-3 (Systems : ‘QA1’/’’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’) top 20 answers returned by the model is sent for post processing and in test batch 4 and 5 only top 5 answers are sent for post processing  System UNCC_QA2(in batch 3) for List type question answering, Context is generated from documents for which URLS are provided in the BioASQ test data  for the rest of the systems (in test batch-3) for List Type question answering task snippets present in the BioaSQ test data are used to generate context 
In test batch-4 (System : ‘FACTOIDS’/’UNCC_QA_1’/’UNCC_QA3’’) top 5 answers returned by the model is sent for post processing  In case of system ‘FACTOIDS’ snippets in the test data were used to generate context  for systems ’UNCC_QA_1’ and ’UNCC_QA3’ context is generated from the documents for which URLS are provided in the BioASQ test data 
In test batch-5 ( Systems: ‘QA1’/’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’ ) our approach is the same as that of test batch-4 where top 5 answers returned by the model is sent for post processing  for all the systems (in test batch-5) context is generated from the snippets provided in the BioASQ test data 
APPENDIX ::: Systems and their descriptions: ::: Yes/No Type Questions:
For the first 3 test batches, We have submitted answer ‘Yes’ to all the questions  Later, we employed ‘Sentence Entailment’ techniques(refer section 6 0) for the fourth and fifth test batch sets  Our Systems with ‘Sentence Entailment’ approach (for ‘Yes’/ ‘No’ question answering): ‘UNCC_QA_1’(test batch-4), UNCC_QA3(test batch-5) 
APPENDIX ::: Additional details for Yes/No Type Questions
We used Textual Entailment in Batch 4 and 5 for ‘Yes’/‘No’ question type  The algorithm was very simple: Given a question we iterate through the candidate sentences, and look for any candidate sentences contradicting the question  If we find one 'No' is returned as answer, else 'Yes' is returned  (The confidence for contradiction was set at 50%) We used AllenNLP BIBREF13 entailment library to find entailment of the candidate sentences with question 
Flow Chart for Yes/No Question answer processing is shown in Fig FIGREF51
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions
There are different question types, and we distinguished them based on the question words: ‘Which’, ‘What’, ‘When’, ‘How’ etc  Each type of question is being handled differently and there are commonalities among the rules written for different question types  How are question words identified? question words have parts of speech(POS): 'WDT', 'WRB', 'WP' 
Assumptions:
1) Lexical answer type (‘LAT’) or focus word is of type Noun and follows the question word 
2) The LAT word is a Subject  (This clearly not always true, but we used a very simple method)  Note: ‘StanfordNLP’ dependency parsing tag for identifying subject is 'nsubj' or 'nsubjpass' 
3) When a question has multiple words that are of type Subject (and Noun), a word that is in proximity to the question word is considered as ‘LAT’ 
4) For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself that is, ‘When’, ‘Who’, ‘Why’ respectively 
Rules and logic flow to traverse a question: The three cases below describe the logic flow of finding LATs  The figures show the grammatical structures used for this purpose 
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-1:
Question with question word ‘How’ 
For questions with question word 'How', the adjective that follows the question word is considered as ‘LAT’ (need not follow immediately)  If an adjective is absent, word 'How' is considered as ‘LAT’  When there are multiple words that are adjectives, a word in close proximity to the question word and follows it is returned as ‘LAT’  Note: The part of speech tag to identify adjectives is 'JJ'  For Other possible question words like ‘whose’  ‘LAT’/Focus word is question words itself 
Example Question: How many selenoproteins are encoded in the human genome?
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-2:
Questions with question words ‘Which’ , ‘What’ and all other possible question words; a 'Noun' immediately following the question word 
Example Question: Which enzyme is targeted by Evolocumab?
Here, Focus word/LAT is ‘enzyme’ which is both Noun and Subject and immediately follows the question word 
When the word immediately following the question word is a noun, the window size is set to ‘3’  This size ‘3’ means that we iterate through the next ‘3’ words (if present) to check if any of the word is both 'Noun' and 'Subject', If so, the word is considered as ‘LAT’/Focus Word  Else the word that is present very next to the question word is considered as ‘LAT’ 
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-3:
Questions with question words ‘Which’ , ‘What’ and all other possible question words; word immediately following the question word is not a 'Noun' 
Example Question: What is the function of the protein Magt1?
Here, Focus word/LAT is ‘function ’ which is both Noun and Subject and does not immediately follow the question word 
When the very next word following the question word is not a Noun, window size is set to ‘5’  Window size ‘5’ corresponds that we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject  If present, the word is considered as ‘LAT’  Else, the 'Noun' close proximity to the question word and follows it is returned as ‘LAT’ 
Ad we mentioned earlier, the accuracy for ‘LAT’ derivation is 75 percent  But clearly the simple logic described above can be improved, as shown in BIBREF9, BIBREF10  Whether this in turn produces improvements in this particular task is an open question 
APPENDIX ::: Proposing Future Experiments
In the current model, we have a shallow neural network with a softmax layer for predicting answer span  Shallow networks however are not good at generalizations  In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span  The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network  The mentioned dense layered question answering Neural network need to be tuned for finding right hyper parameters  An example of such architecture is shown in Fig FIGREF30 
In another experiment we would like to only feed contextual word embeddings for Focus word/ ‘LAT’, paragraph/ Context as input to the question answering neural network  In this experiment we would neglect all embeddings for the question text except that of Focus word/ ‘LAT’  Our assumption and idea for considering focus word and neglecting remaining words in the question is that during training phase it would make more precise for the model to identify the focus of the question and map answers against the question’s focus  To validate our assumption, we would like to take sample question answering data and find the cosine distance between contextual embedding of Focus word and that of the actual answer and verify if the cosine distance is comparatively low in most of the cases 
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network  By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy  Adding ‘LAT’ feature this way instead of feeding Focus word’s word piece embedding directly (as we did in our above experiments) to the BioBERT would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'  Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy 

ground_truth: For Article-Entity placement, they consider two baselines: the first one using only salience-based features, and the second baseline checks if the entity appears in the title of the article. 

For Article-Section Placement, they consider two baselines: the first picks the section with the highest lexical similarity to the article, and the second one picks the most frequent section.

B1. The first baseline uses only the salience-based features by Dunietz and Gillick BIBREF11 ., B2. The second baseline assigns the value relevant to a pair INLINEFORM0 , if and only if INLINEFORM1 appears in the title of INLINEFORM2 .

, S1: Pick the section from template INLINEFORM0 with the highest lexical similarity to INLINEFORM1 : S1 INLINEFORM2, S2: Place the news into the most frequent section in INLINEFORM0

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was their highest recall score?
Context 4:  Interestingly, we found that while masking before convolution gives a lower WER, using masks for both convolutions and batch norm results in worse performance 
As a final note, we found that training with weight norm was very unstable leading to exploding activations 
Residual Connections
For models deeper than Jasper 5x3, we observe consistently that residual connections are necessary for training to converge  In addition to the simple residual and dense residual model described above, we investigated DenseNet BIBREF15 and DenseRNet BIBREF16 variants of Jasper  Both connect the outputs of each sub-block to the inputs of following sub-blocks within a block  DenseRNet, similar to Dense Residual, connects the output of each output of each block to the input of all following blocks  DenseNet and DenseRNet combine residual connections using concatenation whereas Residual and Dense Residual use addition  We found that Dense Residual and DenseRNet perform similarly with each performing better on specific subsets of LibriSpeech  We decided to use Dense Residual for subsequent experiments  The main reason is that due to concatenation, the growth factor for DenseNet and DenseRNet requires tuning for deeper models whereas Dense Residual simply just repeats a sub-blocks 
Language Model
A language model (LM) is a probability distribution over arbitrary symbol sequences INLINEFORM0 such that more likely sequences are assigned high probabilities  LMs are frequently used to condition beam search  During decoding, candidates are evaluated using both acoustic scores and LM scores  Traditional N-gram LMs have been augmented with neural LMs in recent work BIBREF20 , BIBREF21 , BIBREF22  
We experiment with statistical N-gram language models BIBREF23 and neural Transformer-XL BIBREF11 models  Our best results use acoustic and word-level N-gram language models to generate a candidate list using beam search with a width of 2048  Next, an external Transformer-XL LM rescores the final list  All LMs were trained on datasets independently from acoustic models  We show results with the neural LM in our Results section  We observed a strong correlation between the quality of the neural LM (measured by perplexity) and WER as shown in Figure FIGREF20  
NovoGrad
For training, we use either Stochastic Gradient Descent (SGD) with momentum or our own NovoGrad, an optimizer similar to Adam BIBREF14 , except that its second moments are computed per layer instead of per weight  Compared to Adam, it reduces memory consumption and we find it to be more numerically stable 
At each step INLINEFORM0 , NovoGrad computes the stochastic gradient INLINEFORM1 following the regular forward-backward pass  Then the second-order moment INLINEFORM2 is computed for each layer INLINEFORM3 similar to ND-Adam BIBREF27 : DISPLAYFORM0
The second-order moment INLINEFORM0 is used to re-scale gradients INLINEFORM1 before calculating the first-order moment INLINEFORM2 : DISPLAYFORM0
If L2-regularization is used, a weight decay INLINEFORM0 is added to the re-scaled gradient (as in AdamW BIBREF28 ): DISPLAYFORM0
Finally, new weights are computed using the learning rate INLINEFORM0 : DISPLAYFORM0
Using NovoGrad instead of SGD with momentum, we decreased the WER on dev-clean LibriSpeech from 4 00% to 3 64%, a relative improvement of 9% for Jasper DR 10x5  We will further analyze NovoGrad in forthcoming work 
Results
We evaluate Jasper across a number of datasets in various domains  In all experiments, we use dropout and weight decay as regularization  At training time, we use speed perturbation with fixed +/-10% BIBREF29 for LibriSpeech  For WSJ and Hub5'00, we use a random speed perturbation factor between [-10%, 10%] as each utterance is fed into the model  All models have been trained on NVIDIA DGX-1 in mixed precision BIBREF30 using OpenSeq2Seq BIBREF31   Source code, training configurations, and pretrained models are available 
Read Speech
We evaluated the performance of Jasper on two read speech datasets: LibriSpeech and Wall Street Journal (WSJ)  For LibriSpeech, we trained Jasper DR 10x5 using our NovoGrad optimizer for 400 epochs  We achieve SOTA performance on the test-clean subset and SOTA among end-to-end speech recognition models on test-other 
We trained a smaller Jasper 10x3 model with SGD with momentum optimizer for 400 epochs on a combined WSJ dataset (80 hours): LDC93S6A (WSJ0) and LDC94S13A (WSJ1)  The results are provided in Table TABREF29  
Conversational Speech
We also evaluate the Jasper model's performance on a conversational English corpus  The Hub5 Year 2000 (Hub5'00) evaluation (LDC2002S09, LDC2005S13) is widely used in academia  It is divided into two subsets: Switchboard (SWB) and Callhome (CHM)  The training data for both the acoustic and language models consisted of the 2000hr Fisher+Switchboard training data (LDC2004S13, LDC2005S13, LDC97S62)  Jasper DR 10x5 was trained using SGD with momentum for 50 epochs  We compare to other models trained using the same data and report Hub5'00 results in Table TABREF31  
We obtain good results for SWB  However, there is work to be done to improve WER on harder tasks such as CHM 
Conclusions
We have presented a new family of neural architectures for end-to-end speech recognition  Inspired by wav2letter's convolutional approach, we build a deep and scalable model, which requires a well-designed residual topology, effective regularization, and a strong optimizer  As our architecture studies demonstrated, a combination of standard components leads to SOTA results on LibriSpeech and competitive results on other benchmarks  Our Jasper architecture is highly efficient for training and inference, and serves as a good baseline approach on top of which to explore more sophisticated regularization, data augmentation, loss functions, language models, and optimization strategies  We are interested to see if our approach can continue to scale to deeper models and larger datasets 

ground_truth: 0.7033

0.7033