prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which languages are similar to each other?
Context 1: Introduction
Accurate language identification (LID) is the first step in many natural language processing and machine comprehension pipelines  If the language of a piece of text is known then the appropriate downstream models like parts of speech taggers and language models can be applied as required 
LID is further also an important step in harvesting scarce language resources  Harvested data can be used to bootstrap more accurate LID models and in doing so continually improve the quality of the harvested data  Availability of data is still one of the big roadblocks for applying data driven approaches like supervised machine learning in developing countries 
Having 11 official languages of South Africa has lead to initiatives (discussed in the next section) that have had positive effect on the availability of language resources for research  However, many of the South African languages are still under resourced from the point of view of building data driven models for machine comprehension and process automation 
Table TABREF2 shows the percentages of first language speakers for each of the official languages of South Africa  These are four conjunctively written Nguni languages (zul, xho, nbl, ssw), Afrikaans (afr) and English (eng), three disjunctively written Sotho languages (nso, sot, tsn), as well as tshiVenda (ven) and Xitsonga (tso)  The Nguni languages are similar to each other and harder to distinguish  The same is true of the Sotho languages 
This paper presents a hierarchical naive Bayesian and lexicon based classifier for LID of short pieces of text of 15-20 characters long  The algorithm is evaluated against recent approaches using existing test sets from previous works on South African languages as well as the Discriminating between Similar Languages (DSL) 2015 and 2017 shared tasks 
Section SECREF2 reviews existing works on the topic and summarises the remaining research problems  Section SECREF3 of the paper discusses the proposed algorithm and Section SECREF4 presents comparative results 
Related Works
The focus of this section is on recently published datasets and LID research applicable to the South African context  An in depth survey of algorithms, features, datasets, shared tasks and evaluation methods may be found in BIBREF0 
The datasets for the DSL 2015 & DSL 2017 shared tasks BIBREF1 are often used in LID benchmarks and also available on Kaggle   The DSL datasets, like other LID datasets, consists of text sentences labelled by language  The 2017 dataset, for example, contains 14 languages over 6 language groups with 18000 training samples and 1000 testing samples per language 
The recently published JW300 parallel corpus BIBREF2 covers over 300 languages with around 100 thousand parallel sentences per language pair on average  In South Africa, a multilingual corpus of academic texts produced by university students with different mother tongues is being developed BIBREF3  The WiLI-2018 benchmark dataset BIBREF4 for monolingual written natural language identification includes around 1000 paragraphs of 235 languages  A possibly useful link can also be made BIBREF5 between Native Language Identification (NLI) (determining the native language of the author of a text) and Language Variety Identification (LVI) (classification of different varieties of a single language) which opens up more datasets  The Leipzig Corpora Collection BIBREF6, the Universal Declaration of Human Rights and Tatoeba are also often used sources of data 
The NCHLT text corpora BIBREF7 is likely a good starting point for a shared LID task dataset for the South African languages BIBREF8  The NCHLT text corpora contains enough data to have 3500 training samples and 600 testing samples of 300+ character sentences per language  Researchers have recently started applying existing algorithms for tasks like neural machine translation in earnest to such South African language datasets BIBREF9 
Existing NLP datasets, models and services BIBREF10 are available for South African languages  These include an LID algorithm BIBREF11 that uses a character level n-gram language model  Multiple papers have shown that 'shallow' naive Bayes classifiers BIBREF12, BIBREF8, BIBREF13, BIBREF14, SVMs BIBREF15 and similar models work very well for doing LID  The DSL 2017 paper BIBREF1, for example, gives an overview of the solutions of all of the teams that competed on the shared task and the winning approach BIBREF16 used an SVM with character n-gram, parts of speech tag features and some other engineered features  The winning approach for DSL 2015 used an ensemble naive Bayes classifier  The fasttext classifier BIBREF17 is perhaps one of the best known efficient 'shallow' text classifiers that have been used for LID  
Multiple papers have proposed hierarchical stacked classifiers (including lexicons) that would for example first classify a piece of text by language group and then by exact language BIBREF18, BIBREF19, BIBREF8, BIBREF0  Some work has also been done on classifying surnames between Tshivenda, Xitsonga and Sepedi BIBREF20  Additionally, data augmentation BIBREF21 and adversarial training BIBREF22 approaches are potentially very useful to reduce the requirement for data 
Researchers have investigated deeper LID models like bidirectional recurrent neural networks BIBREF23 or ensembles of recurrent neural networks BIBREF24  The latter is reported to achieve 95 12% in the DSL 2015 shared task  In these models text features can include character and word n-grams as well as informative character and word-level features learnt BIBREF25 from the training data  The neural methods seem to work well in tasks where more training data is available 
In summary, LID of short texts, informal styles and similar languages remains a difficult problem which is actively being researched  Increased confusion can in general be expected between shorter pieces of text and languages that are more closely related  Shallow methods still seem to work well compared to deeper models for LID  Other remaining research opportunities seem to be data harvesting, building standardised datasets and creating shared tasks for South Africa and Africa  Support for language codes that include more languages seems to be growing and discoverability of research is improving with more survey papers coming out  Paywalls also seem to no longer be a problem; the references used in this paper was either openly published or available as preprint papers 
Methodology
The proposed LID algorithm builds on the work in BIBREF8 and BIBREF26  We apply a naive Bayesian classifier with character (2, 4 & 6)-grams, word unigram and word bigram features with a hierarchical lexicon based classifier 
The naive Bayesian classifier is trained to predict the specific language label of a piece of text, but used to first classify text as belonging to either the Nguni family, the Sotho family, English, Afrikaans, Xitsonga or Tshivenda  The scikit-learn multinomial naive Bayes classifier is used for the implementation with an alpha smoothing value of 0 01 and hashed text features 
The lexicon based classifier is then used to predict the specific language within a language group  For the South African languages this is done for the Nguni and Sotho groups  If the lexicon prediction of the specific language has high confidence then its result is used as the final label else the naive Bayesian classifier's specific language prediction is used as the final result  The lexicon is built over all the data and therefore includes the vocabulary from both the training and testing sets 
The lexicon based classifier is designed to trade higher precision for lower recall  The proposed implementation is considered confident if the number of words from the winning language is at least one more than the number of words considered to be from the language scored in second place 
The stacked classifier is tested against three public LID implementations BIBREF17, BIBREF23, BIBREF8  The LID implementation described in BIBREF17 is available on GitHub and is trained and tested according to a post on the fasttext blog  Character (5-6)-gram features with 16 dimensional vectors worked the best  The implementation discussed in BIBREF23 is available from https://github com/tomkocmi/LanideNN  Following the instructions for an OSX pip install of an old r0 8 release of TensorFlow, the LanideNN code could be executed in Python 3 7 4  Settings were left at their defaults and a learning rate of 0 001 was used followed by a refinement with learning rate of 0 0001  Only one code modification was applied to return the results from a method that previously just printed to screen  The LID algorithm described in BIBREF8 is also available on GitHub 
The stacked classifier is also tested against the results reported for four other algorithms BIBREF16, BIBREF26, BIBREF24, BIBREF15  All the comparisons are done using the NCHLT BIBREF7, DSL 2015 BIBREF19 and DSL 2017 BIBREF1 datasets discussed in Section SECREF2 
Results and Analysis
The average classification accuracy results are summarised in Table TABREF9  The accuracies reported are for classifying a piece of text by its specific language label  Classifying text only by language group or family is a much easier task as reported in BIBREF8 
Different variations of the proposed classifier were evaluated  A single NB classifier (NB), a stack of two NB classifiers (NB+NB), a stack of a NB classifier and lexicon (NB+Lex) and a lexicon (Lex) by itself  A lexicon with a 50% training token dropout is also listed to show the impact of the lexicon support on the accuracy 
From the results it seems that the DSL 2017 task might be harder than the DSL 2015 and NCHLT tasks  Also, the results for the implementation discussed in BIBREF23 might seem low, but the results reported in that paper is generated on longer pieces of text so lower scores on the shorter pieces of text derived from the NCHLT corpora is expected 
The accuracy of the proposed algorithm seems to be dependent on the support of the lexicon  Without a good lexicon a non-stacked naive Bayesian classifier might even perform better 
The execution performance of some of the LID implementations are shown in Table TABREF10  Results were generated on an early 2015 13-inch Retina MacBook Pro with a 2 9 GHz CPU (Turbo Boosted to 3 4 GHz) and 8GB RAM  The C++ implementation in BIBREF17 is the fastest  The implementation in BIBREF8 makes use of un-hashed feature representations which causes it to be slower than the proposed sklearn implementation  The execution performance of BIBREF23 might improve by a factor of five to ten when executed on a GPU 
Conclusion
LID of short texts, informal styles and similar languages remains a difficult problem which is actively being researched  The proposed algorithm was evaluated on three existing datasets and compared to the implementations of three public LID implementations as well as to reported results of four other algorithms  It performed well relative to the other methods beating their results  However, the performance is dependent on the support of the lexicon 
We would like to investigate the value of a lexicon in a production system and how to possibly maintain it using self-supervised learning  We are investigating the application of deeper language models some of which have been used in more recent DSL shared tasks  We would also like to investigate data augmentation strategies to reduce the amount of training data that is required 
Further research opportunities include data harvesting, building standardised datasets and shared tasks for South Africa as well as the rest of Africa  In general, the support for language codes that include more languages seems to be growing, discoverability of research is improving and paywalls seem to no longer be a big problem in getting access to published research 
Context 2: Introduction
The task of generating natural language descriptions of structured data (such as tables) BIBREF2 , BIBREF3 , BIBREF4 has seen a growth in interest with the rise of sequence to sequence models that provide an easy way of encoding tables and generating text from them BIBREF0 , BIBREF1 , BIBREF5 , BIBREF6  
For text generation tasks, the only gold standard metric is to show the output to humans for judging its quality, but this is too expensive to apply repeatedly anytime small modifications are made to a system  Hence, automatic metrics that compare the generated text to one or more reference texts are routinely used to compare models BIBREF7   For table-to-text generation, automatic evaluation has largely relied on BLEU BIBREF8 and ROUGE BIBREF9   The underlying assumption behind these metrics is that the reference text is gold-standard, i e , it is the ideal target text that a system should generate  In practice, however, when datasets are collected automatically and heuristically, the reference texts are often not ideal  Figure FIGREF2 shows an example from the WikiBio dataset BIBREF0   Here the reference contains extra information which no system can be expected to produce given only the associated table  We call such reference texts divergent from the table 
We show that existing automatic metrics, including BLEU, correlate poorly with human judgments when the evaluation sets contain divergent references (§ SECREF36 )  For many table-to-text generation tasks, the tables themselves are in a pseudo-natural language format (e g , WikiBio, WebNLG BIBREF6 , and E2E-NLG BIBREF10 )  In such cases we propose to compare the generated text to the underlying table as well to improve evaluation  We develop a new metric, PARENT (Precision And Recall of Entailed N-grams from the Table) (§ SECREF3 )  When computing precision, PARENT effectively uses a union of the reference and the table, to reward correct information missing from the reference  When computing recall, it uses an intersection of the reference and the table, to ignore extra incorrect information in the reference  The union and intersection are computed with the help of an entailment model to decide if a text n-gram is entailed by the table  We show that this method is more effective than using the table as an additional reference  Our main contributions are:
Table-to-Text Generation
We briefly review the task of generating natural language descriptions of semi-structured data, which we refer to as tables henceforth BIBREF11 , BIBREF12   Tables can be expressed as set of records INLINEFORM0 , where each record is a tuple (entity, attribute, value)  When all the records are about the same entity, we can truncate the records to (attribute, value) pairs  For example, for the table in Figure FIGREF2 , the records are {(Birth Name, Michael Dahlquist), (Born, December 22 1965),    }  The task is to generate a text INLINEFORM1 which summarizes the records in a fluent and grammatical manner  For training and evaluation we further assume that we have a reference description INLINEFORM2 available for each table  We let INLINEFORM3 denote an evaluation set of tables, references and texts generated from a model INLINEFORM4 , and INLINEFORM5 , INLINEFORM6 denote the collection of n-grams of order INLINEFORM7 in INLINEFORM8 and INLINEFORM9 , respectively  We use INLINEFORM10 to denote the count of n-gram INLINEFORM11 in INLINEFORM12 , and INLINEFORM13 to denote the minimum of its counts in INLINEFORM14 and INLINEFORM15   Our goal is to assign a score to the model, which correlates highly with human judgments of the quality of that model 
PARENT
PARENT evaluates each instance INLINEFORM0 separately, by computing the precision and recall of INLINEFORM1 against both INLINEFORM2 and INLINEFORM3  
Evaluation via Information Extraction
BIBREF1 proposed to use an auxiliary model, trained to extract structured records from text, for evaluation  However, the extraction model presented in that work is limited to the closed-domain setting of basketball game tables and summaries  In particular, they assume that each table has exactly the same set of attributes for each entity, and that the entities can be identified in the text via string matching  These assumptions are not valid for the open-domain WikiBio dataset, and hence we train our own extraction model to replicate their evaluation scheme 
Our extraction system is a pointer-generator network BIBREF19 , which learns to produce a linearized version of the table from the text  The network learns which attributes need to be populated in the output table, along with their values  It is trained on the training set of WikiBio  At test time we parsed the output strings into a set of (attribute, value) tuples and compare it to the ground truth table  The F-score of this text-to-table system was INLINEFORM0 , which is comparable to other challenging open-domain settings BIBREF20   More details are included in the Appendix SECREF52  
Given this information extraction system, we consider the following metrics for evaluation, along the lines of BIBREF1   Content Selection (CS): F-score for the (attribute, value) pairs extracted from the generated text compared to those extracted from the reference  Relation Generation (RG): Precision for the (attribute, value) pairs extracted from the generated text compared to those in the ground truth table  RG-F: Since our task emphasizes the recall of information from the table as well, we consider another variant which computes the F-score of the extracted pairs to those in the table  We omit the content ordering metric, since our extraction system does not align records to the input text 
Experiments & Results
In this section we compare several automatic evaluation metrics by checking their correlation with the scores assigned by humans to table-to-text models  Specifically, given INLINEFORM0 models INLINEFORM1 , and their outputs on an evaluation set, we show these generated texts to humans to judge their quality, and obtain aggregated human evaluation scores for all the models, INLINEFORM2 (§ SECREF33 )  Next, to evaluate an automatic metric, we compute the scores it assigns to each model, INLINEFORM3 , and check the Pearson correlation between INLINEFORM4 and INLINEFORM5 BIBREF21  
Data & Models
Our main experiments are on the WikiBio dataset BIBREF0 , which is automatically constructed and contains many divergent references  In § SECREF47 we also present results on the data released as part of the WebNLG challenge 
We developed several models of varying quality for generating text from the tables in WikiBio  This gives us a diverse set of outputs to evaluate the automatic metrics on  Table TABREF32 lists the models along with their hyperparameter settings and their scores from the human evaluation (§ SECREF33 )  Our focus is primarily on neural sequence-to-sequence methods since these are most widely used, but we also include a template-based baseline  All neural models were trained on the WikiBio training set  Training details and sample outputs are included in Appendices SECREF56 & SECREF57  
We divide these models into two categories and measure correlation separately for both the categories  The first category, WikiBio-Systems, includes one model each from the four families listed in Table TABREF32   This category tests whether a metric can be used to compare different model families with a large variation in the quality of their outputs  The second category, WikiBio-Hyperparams, includes 13 different hyperparameter settings of PG-Net BIBREF19 , which was the best performing system overall  9 of these were obtained by varying the beam size and length normalization penalty of the decoder network BIBREF23 , and the remaining 4 were obtained by re-scoring beams of size 8 with the information extraction model described in § SECREF4   All the models in this category produce high quality fluent texts, and differ primarily on the quantity and accuracy of the information they express  Here we are testing whether a metric can be used to compare similar systems with a small variation in performance  This is an important use-case as metrics are often used to tune hyperparameters of a model 
Human Evaluation
We collected human judgments on the quality of the 16 models trained for WikiBio, plus the reference texts  Workers on a crowd-sourcing platform, proficient in English, were shown a table with pairs of generated texts, or a generated text and the reference, and asked to select the one they prefer  Figure FIGREF34 shows the instructions they were given  Paired comparisons have been shown to be superior to rating scales for comparing generated texts BIBREF24   However, for measuring correlation the comparisons need to be aggregated into real-valued scores, INLINEFORM0 , for each of the INLINEFORM1 models  For this, we use Thurstone's method BIBREF22 , which assigns a score to each model based on how many times it was preferred over an alternative 
The data collection was performed separately for models in the WikiBio-Systems and WikiBio-Hyperparams categories  1100 tables were sampled from the development set, and for each table we got 8 different sentence pairs annotated across the two categories, resulting in a total of 8800 pairwise comparisons  Each pair was judged by one worker only which means there may be noise at the instance-level, but the aggregated system-level scores had low variance (cf  Table TABREF32 )  In total around 500 different workers were involved in the annotation  References were also included in the evaluation, and they received a lower score than PG-Net, highlighting the divergence in WikiBio 
Compared Metrics
Text only: We compare BLEU BIBREF8 , ROUGE BIBREF9 , METEOR BIBREF18 , CIDEr and CIDEr-D BIBREF25 using their publicly available implementations 
Information Extraction based: We compare the CS, RG and RG-F metrics discussed in § SECREF4  
Text & Table: We compare a variant of BLEU, denoted as BLEU-T, where the values from the table are used as additional references  BLEU-T draws inspiration from iBLEU BIBREF26 but instead rewards n-grams which match the table rather than penalizing them  For PARENT, we compare both the word-overlap model (PARENT-W) and the co-occurrence model (PARENT-C) for determining entailment  We also compare versions where a single INLINEFORM0 is tuned on the entire dataset to maximize correlation with human judgments, denoted as PARENT*-W/C 
Correlation Comparison
We use bootstrap sampling (500 iterations) over the 1100 tables for which we collected human annotations to get an idea of how the correlation of each metric varies with the underlying data  In each iteration, we sample with replacement, tables along with their references and all the generated texts for that table  Then we compute aggregated human evaluation and metric scores for each of the models and compute the correlation between the two  We report the average correlation across all bootstrap samples for each metric in Table TABREF37   The distribution of correlations for the best performing metrics are shown in Figure FIGREF38  
Table TABREF37 also indicates whether PARENT is significantly better than a baseline metric  BIBREF21 suggest using the William's test for this purpose, but since we are computing correlations between only 4/13 systems at a time, this test has very weak power in our case  Hence, we use the bootstrap samples to obtain a INLINEFORM0 confidence interval of the difference in correlation between PARENT and any other metric and check whether this is above 0 BIBREF27  
Correlations are higher for the systems category than the hyperparams category  The latter is a more difficult setting since very similar models are compared, and hence the variance of the correlations is also high  Commonly used metrics which only rely on the reference (BLEU, ROUGE, METEOR, CIDEr) have only weak correlations with human judgments  In the hyperparams category, these are often negative, implying that tuning models based on these may lead to selecting worse models  BLEU performs the best among these, and adding n-grams from the table as references improves this further (BLEU-T) 
Among the extractive evaluation metrics, CS, which also only relies on the reference, has poor correlation in the hyperparams category  RG-F, and both variants of the PARENT metric achieve the highest correlation for both settings  There is no significant difference among these for the hyperparams category, but for systems, PARENT-W is significantly better than the other two  While RG-F needs a full information extraction pipeline in its implementation, PARENT-C only relies on co-occurrence counts, and PARENT-W can be used out-of-the-box for any dataset  To our knowledge, this is the first rigorous evaluation of using information extraction for generation evaluation 
On this dataset, the word-overlap model showed higher correlation than the co-occurrence model for entailment  In § SECREF47 we will show that for the WebNLG dataset, where more paraphrasing is involved between the table and text, the opposite is true  Lastly, we note that the heuristic for selecting INLINEFORM0 is sufficient to produce high correlations for PARENT, however, if human annotations are available, this can be tuned to produce significantly higher correlations (PARENT*-W/C) 
Analysis
In this section we further analyze the performance of PARENT-W under different conditions, and compare to the other best metrics from Table TABREF37  
To study the correlation as we vary the number of divergent references, we also collected binary labels from workers for whether a reference is entailed by the corresponding table  We define a reference as entailed when it mentions only information which can be inferred from the table  Each table and reference pair was judged by 3 independent workers, and we used the majority vote as the label for that pair  Overall, only INLINEFORM0 of the references were labeled as entailed by the table  Fleiss' INLINEFORM1 was INLINEFORM2 , which indicates a fair agreement  We found the workers sometimes disagreed on what information can be reasonably entailed by the table 
Figure FIGREF40 shows the correlations as we vary the percent of entailed examples in the evaluation set of WikiBio  Each point is obtained by fixing the desired proportion of entailed examples, and sampling subsets from the full set which satisfy this proportion  PARENT and RG-F remain stable and show a high correlation across the entire range, whereas BLEU and BLEU-T vary a lot  In the hyperparams category, the latter two have the worst correlation when the evaluation set contains only entailed examples, which may seem surprising  However, on closer examination we found that this subset tends to omit a lot of information from the tables  Systems which produce more information than these references are penalized by BLEU, but not in the human evaluation  PARENT overcomes this issue by measuring recall against the table in addition to the reference 
We check how different components in the computation of PARENT contribute to its correlation to human judgments  Specifically, we remove the probability INLINEFORM0 of an n-gram INLINEFORM1 being entailed by the table from Eqs  EQREF19 and EQREF23   The average correlation for PARENT-W drops to INLINEFORM5 in this case  We also try a variant of PARENT with INLINEFORM6 , which removes the contribution of Table Recall (Eq  EQREF22 )  The average correlation is INLINEFORM7 in this case  With these components, the correlation is INLINEFORM8 , showing that they are crucial to the performance of PARENT 
BIBREF28 point out that hill-climbing on an automatic metric is meaningless if that metric has a low instance-level correlation to human judgments  In Table TABREF46 we show the average accuracy of the metrics in making the same judgments as humans between pairs of generated texts  Both variants of PARENT are significantly better than the other metrics, however the best accuracy is only INLINEFORM0 for the binary task  This is a challenging task, since there are typically only subtle differences between the texts  Achieving higher instance-level accuracies will require more sophisticated language understanding models for evaluation 
WebNLG Dataset
To check how PARENT correlates with human judgments when the references are elicited from humans (and less likely to be divergent), we check its correlation with the human ratings provided for the systems competing in the WebNLG challenge BIBREF6   The task is to generate text describing 1-5 RDF triples (e g  John E Blaha, birthPlace, San Antonio), and human ratings were collected for the outputs of 9 participating systems on 223 instances  These systems include a mix of pipelined, statistical and neural methods  Each instance has upto 3 reference texts associated with the RDF triples, which we use for evaluation 
The human ratings were collected on 3 distinct aspects – grammaticality, fluency and semantics, where semantics corresponds to the degree to which a generated text agrees with the meaning of the underlying RDF triples  We report the correlation of several metrics with these ratings in Table TABREF48   Both variants of PARENT are either competitive or better than the other metrics in terms of the average correlation to all three aspects  This shows that PARENT is applicable for high quality references as well 
While BLEU has the highest correlation for the grammar and fluency aspects, PARENT does best for semantics  This suggests that the inclusion of source tables into the evaluation orients the metric more towards measuring the fidelity of the content of the generation  A similar trend is seen comparing BLEU and BLEU-T  As modern neural text generation systems are typically very fluent, measuring their fidelity is of increasing importance  Between the two entailment models, PARENT-C is better due to its higher correlation with the grammaticality and fluency aspects 
The INLINEFORM0 parameter in the calculation of PARENT decides whether to compute recall against the table or the reference (Eq  EQREF22 )  Figure FIGREF50 shows the distribution of the values taken by INLINEFORM1 using the heuristic described in § SECREF3 for instances in both WikiBio and WebNLG  For WikiBio, the recall of the references against the table is generally low, and hence the recall of the generated text relies more on the table  For WebNLG, where the references are elicited from humans, this recall is much higher (often INLINEFORM2 ), and hence the recall of the generated text relies more on the reference 
Related Work
Over the years several studies have evaluated automatic metrics for measuring text generation performance BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35   The only consensus from these studies seems to be that no single metric is suitable across all tasks  A recurring theme is that metrics like BLEU and NIST BIBREF36 are not suitable for judging content quality in NLG  Recently, BIBREF37 did a comprehensive study of several metrics on the outputs of state-of-the-art NLG systems, and found that while they showed acceptable correlation with human judgments at the system level, they failed to show any correlation at the sentence level  Ours is the first study which checks the quality of metrics when table-to-text references are divergent  We show that in this case even system level correlations can be unreliable 
Hallucination BIBREF38 , BIBREF39 refers to when an NLG system generates text which mentions extra information than what is present in the source from which it is generated  Divergence can be viewed as hallucination in the reference text itself  PARENT deals with hallucination by discounting n-grams which do not overlap with either the reference or the table 
PARENT draws inspiration from iBLEU BIBREF26 , a metric for evaluating paraphrase generation, which compares the generated text to both the source text and the reference  While iBLEU penalizes texts which match the source, here we reward such texts since our task values accuracy of generated text more than the need for paraphrasing the tabular content BIBREF40   Similar to SARI for text simplification BIBREF41 and Q-BLEU for question generation BIBREF42 , PARENT falls under the category of task-specific metrics 
Conclusions
We study the automatic evaluation of table-to-text systems when the references diverge from the table  We propose a new metric, PARENT, which shows the highest correlation with humans across a range of settings with divergent references in WikiBio  We also perform the first empirical evaluation of information extraction based metrics BIBREF1 , and find RG-F to be effective  Lastly, we show that PARENT is comparable to the best existing metrics when references are elicited by humans on the WebNLG data 
Acknowledgements
Bhuwan Dhingra is supported by a fellowship from Siemens, and by grants from Google  We thank Maruan Al-Shedivat, Ian Tenney, Tom Kwiatkowski, Michael Collins, Slav Petrov, Jason Baldridge, David Reitter and other members of the Google AI Language team for helpful discussions and suggestions  We thank Sam Wiseman for sharing data for an earlier version of this paper  We also thank the anonymous reviewers for their feedback 
Information Extraction System
For evaluation via information extraction BIBREF1 we train a model for WikiBio which accepts text as input and generates a table as the output  Tables in WikiBio are open-domain, without any fixed schema for which attributes may be present or absent in an instance  Hence we employ the Pointer-Generator Network (PG-Net) BIBREF19 for this purpose  Specifically, we use a sequence-to-sequence model, whose encoder and decoder are both single-layer bi-directional LSTMs  The decoder is augmented with an attention mechanism over the states of the encoder  Further, it also uses a copy mechanism to optionally copy tokens directly from the source text  We do not use the coverage mechanism of BIBREF19 since that is specific to the task of summarization they study  The decoder is trained to produce a linearized version of the table where the rows and columns are flattened into a sequence, and separate by special tokens  Figure FIGREF53 shows an example 
Clearly, since the references are divergent, the model cannot be expected to produce the entire table, and we see some false information being hallucinated after training  Nevertheless, as we show in § SECREF36 , this system can be used for evaluating generated texts  After training, we can parse the output sequence along the special tokens INLINEFORM0 R INLINEFORM1 and INLINEFORM2 C INLINEFORM3 to get a set of (attribute, value) pairs  Table TABREF54 shows the precision, recall and F-score of these extracted pairs against the ground truth tables, where the attributes and values are compared using an exact string match 
Hyperparameters
After tuning we found the same set of hyperparameters to work well for both the table-to-text PG-Net, and the inverse information extraction PG-Net  The hidden state size of the biLSTMs was set to 200  The input and output vocabularies were set to 50000 most common words in the corpus, with additional special symbols for table attribute names (such as “birth-date”)  The embeddings of the tokens in the vocabulary were initialized with Glove BIBREF43   Learning rate of INLINEFORM0 was used during training, with the Adam optimizer, and a dropout of INLINEFORM1 was also applied to the outputs of the biLSTM  Models were trained till the loss on the dev set stopped dropping  Maximum length of a decoded text was set to 40 tokens, and that of the tables was set to 120 tokens  Various beam sizes and length normalization penalties were applied for the table-to-text system, which are listed in the main paper  For the information extraction system, we found a beam size of 8 and no length penalty to produce the highest F-score on the dev set 
Sample Outputs
Table TABREF55 shows some sample references and the corresponding predictions from the best performing model, PG-Net for WikiBio 
Context 3: Introduction
Part-of-speech tagging is now a classic task in natural language processing, for which many systems have been developed or adapted for a large variety of languages  Its aim is to associate each “word” with a morphosyntactic tag, whose granularity can range from a simple morphosyntactic category, or part-of-speech (hereafter PoS), to finer categories enriched with morphological features (gender, number, case, tense, mood, etc ) 
The use of machine learning algorithms trained on manually annotated corpora has long become the standard way to develop PoS taggers  A large variety of algorithms have been used, such as (in approximative chronological order) bigram and trigram hidden Markov models BIBREF0 , BIBREF1 , BIBREF2 , decision trees BIBREF3 , BIBREF4 , maximum entropy Markov models (MEMMs) BIBREF5 and Conditional Random Fields (CRFs) BIBREF6 , BIBREF7   With such machine learning algorithms, it is possible to build PoS taggers for any language, provided adequate training data is available 
As a complement to annotated corpora, it has previously been shown that external lexicons are valuable sources of information, in particular morphosyntactic lexicons, which provide a large inventory of (word, PoS) pairs  Such lexical information can be used in the form of constraints at tagging time BIBREF8 , BIBREF9 or during the training process as additional features combined with standard features extracted from the training corpus BIBREF10 , BIBREF11 , BIBREF12  
In recent years, a different approach to modelling lexical information and integrating it into natural language processing systems has emerged, namely the use of vector representations for words or word sequences BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18   Such representations, which are generally extracted from large amounts of raw text, have proved very useful for numerous tasks including PoS tagging, in particular when used in recurrent neural networks (RNNs) and more specifically in mono- or bi-directional, word-level and/or character-level long short-term memory networks (LSTMs) BIBREF19 , BIBREF16 , BIBREF17 , BIBREF20  
Both approaches to representing lexical properties and to integrating them into a PoS tagger improve tagging results  Yet they rely on resources of different natures  The main advantage of word vectors is that they are built in an unsupervised way, only requiring large amounts of raw textual data  They also encode finer-grained information than usual morphosyntactic lexicons, most of which do not include any quantitative data, not even simple frequency information  Conversely, lexical resources often provide information about scarcely attested words, for which corpus-based approaches such as word vector representations are of limited relevance  Moreover, morphological or morphosyntactic lexicons already exist for a number of languages, including less-resourced langauges for which it might be difficult to obtain the large amounts of raw data necessary to extract word vector representations 
Our main goal is therefore to compare the respective impact of external lexicons and word vector representations on the accuracy of PoS models  This question has already been investigated for 6 languages by BIBREF18 using the state-of-the-art CRF-based tagging system MarMoT  The authors found that their best-performing word-vector-based PoS tagging models outperform their models that rely on morphosyntactic resources (lexicons or morphological analysers)  In this paper, we report on larger comparison, carried out in a larger multilingual setting and comparing different tagging models  Using different 16 datasets, we compare the performances of two feature-based models enriched with external lexicons and of two LSTM-based models enriched with word vector representations  A secondary goal of our work is to compare the relative improvements linked to the use of external lexical information in the two feature-based models, which use different models (MEMM vs  CRF) and feature sets 
More specifically, our starting point is the MElt system BIBREF12 , an MEMM tagging system  We first briefly describe this system and the way we adapted it by integrating our own set of corpus-based and lexical features  We then introduce the tagging models we have trained for 16 different languages using our adapted version of MElt  These models are trained on the Universal Dependencies (v1 2) corpus set BIBREF21 , complemented by morphosyntactic lexicons  We compare the accuracy of our models with the scores obtained by the CRF-based system MarMoT BIBREF22 , BIBREF18 , retrained on the same corpora and the same external morphosyntactic lexicons  We also compare our results to those obtained by the best bidirectional LSTM models described by BIBREF20 , which both make use of Polyglot word vector representations published by BIBREF23   We will show that an optimised enrichment of feature-based models with morphosyntactic lexicon results in significant accuracy gains  The macro-averaged accuracy of our enriched MElt models is above that of enriched MarMoT models and virtually identical to that of LSTMs enriched with word vector representations  More precisely, per-language results indicate that lexicons provide more useful information for languages with a high lexical variability (such as morphologically rich languages), whereas word vectors are more informative for languages with a lower lexical variability (such as English) 
MElt
MElt BIBREF12 is a tagging system based on maximum entropy Markov models (MEMM) BIBREF5 , a class of discriminative models that are suitable for sequence labelling BIBREF5   The basic set of features used by MElt is given in BIBREF12   It is a superset of the feature sets used by BIBREF5 and BIBREF24 and includes both local standard features (for example the current word itself and its prefixes and suffixes of length 1 to 4) and contextual standard features (for example the tag just assigned to the preceding word)  In particular, with respect to Ratnaparkhi's feature set, MElt's basic feature set lifts the restriction that local standard features used to analyse the internal composition of the current word should only apply to rare words 
One of the advantages of feature-based models such as MEMMs and CRFs is that complementary information can be easily added in the form of additional features  This was investigated for instance by BIBREF25 , whose best-performing model for PoS tagging dialogues was obtained with a version of MElt extended with dialogue-specific features  Yet the motivation of MElt's developers was first and foremost to investigate the best way to integrate lexical information extracted from large-scale morphosyntactic lexical resources into their models, on top of the training data BIBREF12   They showed that performances are better when this external lexical information is integrated in the form of additional lexical features than when the external lexicon is used as constraints at tagging time  These lexical features can also be divided into local lexical features (for example the list of possible tags known to the external lexicon for the current word) and contextual lexical features (for example the list of possible tags known to the external lexicon for surrounding words)  In particular, lexical contextual features provide a means to model the right context of the current word, made of words that have not yet been tagged by the system but for which the lexicon often provides a list of possible tags  Moreover, tagging accuracy for out-of-vocabulary (OOV) words is improved, as a result of the fact that words unknown to the training corpus might be known to the external lexicon 
Despite a few experiments published with MElt on languages other than French BIBREF12 , BIBREF40 , BIBREF41 , the original feature set used by MElt (standard and lexical features) was designed and tested mostly on this language, by building and evaluating tagging models on a variant of the French TreeBank  Since our goal was to carry out experiments in a multilingual setting, we have decided to design our own set of features, using the standard MElt features as a starting point  With respect to the original MElt feature set, we have added new ones, such as prefixes and suffixes of the following word, as well as a hybrid contextual feature obtained by concatenating the tag predicted for the preceding word and the tag(s) provided by the external lexicon for the following word 
In order to select the best performing feature set, we carried out a series of experiments using the multilingual dataset provided during the SPMRL parsing shared task BIBREF42   This included discarding useless or harmful features and selecting the maximal length of the prefixes and suffixes to be used as features, both for the current word and for the following word 
We incorporated in MElt the best performing feature set, described in Table TABREF1   All models discussed in this paper are based on this feature set 
Corpora
We carried out our experiments on the Universal Dependencies v1 2 treebanks BIBREF21 , hereafter UD1 2, from which morphosyntactically annotated corpora can be trivially extracted  All UD1 2 corpora use a common tag set, the 17 universal PoS tags, which is an extension of the tagset proposed by BIBREF43  
As our goal is to study the impact of lexical information for PoS tagging, we have restricted our experiments to UD1 2 corpora that cover languages for which we have morphosyntactic lexicons at our disposal, and for which BIBREF20 provide results  We considered UD1 2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish  Although this language list contains only one non-Indo-European (Indonesian), four major Indo-European sub-families are represented (Germanic, Romance, Slavic, Indo-Iranian)  Overall, the 16 languages considered in our experiments are typologically, morphologically and syntactically fairly diverse 
Lexicons
We generate our external lexicons using the set of source lexicons listed in Table TABREF3   Since external lexical information is exploited via features, there is no need for the external lexicons and the annotated corpora to use the same PoS inventory  Therefore, for each language, we simply extracted from the corresponding lexicon the PoS of each word based on its morphological tags, by removing all information provided except for its coarsest-level category  We also added entries for punctuations when the source lexicons did not contain any 
We also performed experiments in which we retained the full original tags provided by the lexicons, with all morphological features included  On average, results were slightly better than those presented in the paper, although not statistically significantly  Moreover, the granularity of tag inventories in the lexicons is diverse, which makes it difficult to draw general conclusions about results based on full tags  This is why we only report results based on (coarse) PoS extracted from the original lexicons 
Baseline models
In order to assess the respective contributions of external lexicons and word vector representations, we first compared the results of the three above-mentioned systems when trained without such additional lexical information  Table TABREF11 provides the results of MElt and MarMoT retrained on UD1 2 corpora, together with the results publised on the same corpora by BIBREF20 , using their best model not enhanced by external word vector representations —i e  the model they call INLINEFORM0 , which is a bidirectional LSTM that combines both word and character embeddings 
These results show that Plank et al 's (2016) bi-LSTM performs extremely well, surpassed by MarMoT on only 3 out of 16 datasets (Czech, French and Italian), and by MElt only once (Indonesian) 
Models enriched with external lexical information
Table TABREF13 provides the results of four systems enriched with lexical information  The feature-based systems MElt and MarMoT, respectively based on MEMMs and CRFs, are extended with the lexical information provided by our morphosyntactic lexicons  This extension takes the form of additional features, as described in Section SECREF2 for MElt  The results reported by BIBREF20 for their bidirectional LSTM when initialised with Polyglot embeddings trained on full wikipedias are also included, together with their new system FREQBIN, also initialised with Polyglot embeddings  FREQBIN trains bi-LSTMs to predict for each input word both a PoS and a label that represents its log frequency in the training data  As they word it, “the idea behind this model is to make the representation predictive for frequency, which encourages the model not to share representations between common and rare words, thus benefiting the handling of rare tokens ”
The results, which are also displayed in Figures FIGREF14 and FIGREF15 , show that all systems reach very similar results on average, although discrepancies can be observed from one dataset to another, on which we shall comment shortly  The best performing system in terms of macro-average is MElt (96 60%)  Both bi-LSTM systems reach the same score (96 58%), the difference with MElt's results being non significant, whereas MarMoT is only 0 14% behind (96 46%)  Given the better baseline scores of the neural approaches, these results show that the benefit of using external lexicons in the feature-based models MElt and MarMoT are much higher than those using Polyglot word vector representations as initialisations for bi-LSTMs 
Yet these very similar overall results reflect a different picture when focusing on OOV tagging accuracy  The best models for OOV tagging accuracy are, by far, FREQBIN models, which are beaten by MarMoT and by MElt only once each (on English and Danish respectively)  The comparison on OOV tagging between MElt and MarMoT shows that MElt performs better on average than MarMoT, despite the fact that MarMoT's baseline results were better than those reached by MElt  This shows that the information provided by external morphosyntactic lexicons is better exploited by MElt's lexical features than by those used by MarMoT  On the other hand, the comparison of both bi-LSTM-based approaches confirm that the FREQBIN models is better by over 10% absolute on OOV tagging accuracy (94 28% vs  83 59%), with 65% lower error rate 
One of the important differences between the lexical information provided by an external lexicon and word vectors built from raw corpora, apart from the very nature of the lexical information provided, is the coverage and accuracy of this lexical information on rare words  All words in a morphosyntactic lexicon are associated with information of a same granularity and quality, which is not the case with word representations such as provided by Polyglot  Models that take advantage of external lexicons should therefore perform comparatively better on datasets containing a higher proportion of rarer words, provided the lexicons' coverage is high  In order to confirm this intuition, we have used a lexical richness metric based on the type/token ratio  Since this ratio is well-known for being sensitive to corpus length, we normalised it by computing it over the 60,000 first tokens of each training set  When this normalised type/token ratio is plotted against the difference between the results of MElt and both bi-LSTM-based models, the expected correlation is clearly visible (see Figure FIGREF16 )  This explains why MElt obtains better results on the morphologically richer Slavic datasets (average normalised type/token ratio: 0 28, average accuracy difference: 0 32 compared to both bi-LSTM+Polyglot and FREQBIN+Polyglot) and, at the other end of the spectrum, significantly worse results on the English dataset (normalised type/token ratio: 0 15, average accuracy difference: -0 56 compared to bi-LSTM+Polyglot, -0 57 compared to FREQBIN+Polyglot) 
Conclusion
Two main conclusions can be drawn from our comparative results  First, feature-based tagging models adequately enriched with external morphosyntactic lexicons perform, on average, as well as bi-LSTMs enriched with word embeddings  Per-language results show that the best accuracy levels are reached by feature-based models, and in particular by our improved version of the MEMM-based system MElt, on datasets with high lexical variability (in short, for morphologically rich languages), whereas neural-based results perform better on datatsets with lower lexical variability (e g  for English) 
We have only compared the contribution of morphosyntactic lexicons to feature-based models (MEMMs, CRFs) and that of word vector representations to bi-LSTM-based models as reported by BIBREF20   As mentioned above, work on the contribution of word vector representations to feature-based approaches has been carried out by BIBREF18   However, the exploitation of existing morphosyntactic or morphological lexicons in neural models is a less studied question  Improvements over the state of the art might be achieved by integrating lexical information both from an external lexicon and from word vector representations into tagging models 
In that regard, further work will be required to understand which class of models perform the best  An option would be to integrate feature-based models such as a CRF with an LSTM-based layer, following recent proposals such as the one proposed by BIBREF45 for named entity recognition 

ground_truth: Nguni languages (zul, xho, nbl, ssw), Sotho languages (nso, sot, tsn)

The Nguni languages are similar to each other, The same is true of the Sotho languages

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Ngrams of which length are aligned using PARENT?
Context 4: Introduction
We posses a wealth of prior knowledge about many natural language processing tasks  For example, in text categorization, we know that words such as NBA, player, and basketball are strong indicators of the sports category BIBREF0 , and words like terrible, boring, and messing indicate a negative polarity while words like perfect, exciting, and moving suggest a positive polarity in sentiment classification 
A key problem arisen here, is how to leverage such knowledge to guide the learning process, an interesting problem for both NLP and machine learning communities  Previous studies addressing the problem fall into several lines  First, to leverage prior knowledge to label data BIBREF1 , BIBREF2   Second, to encode prior knowledge with a prior on parameters, which can be commonly seen in many Bayesian approaches BIBREF3 , BIBREF4   Third, to formalise prior knowledge with additional variables and dependencies BIBREF5   Last, to use prior knowledge to control the distributions over latent output variables BIBREF6 , BIBREF7 , BIBREF8 , which makes the output variables easily interpretable 
However, a crucial problem, which has rarely been addressed, is the bias in the prior knowledge that we supply to the learning model  Would the model be robust or sensitive to the prior knowledge? Or, which kind of knowledge is appropriate for the task? Let's see an example: we may be a baseball fan but unfamiliar with hockey so that we can provide a few number of feature words of baseball, but much less of hockey for a baseball-hockey classification task  Such prior knowledge may mislead the model with heavy bias to baseball  If the model cannot handle this situation appropriately, the performance may be undesirable 
In this paper, we investigate into the problem in the framework of Generalized Expectation Criteria BIBREF7   The study aims to reveal the factors of reducing the sensibility of the prior knowledge and therefore to make the model more robust and practical  To this end, we introduce auxiliary regularization terms in which our prior knowledge is formalized as distribution over output variables  Recall the example just mentioned, though we do not have enough knowledge to provide features for class hockey, it is easy for us to provide some neutral words, namely words that are not strong indicators of any class, like player here  As one of the factors revealed in this paper, supplying neutral feature words can boost the performance remarkably, making the model more robust 
More attractively, we do not need manual annotation to label these neutral feature words in our proposed approach 
More specifically, we explore three regularization terms to address the problem: (1) a regularization term associated with neutral features; (2) the maximum entropy of class distribution regularization term; and (3) the KL divergence between reference and predicted class distribution  For the first manner, we simply use the most common features as neutral features and assume the neutral features are distributed uniformly over class labels  For the second and third one, we assume we have some knowledge about the class distribution which will be detailed soon later 
To summarize, the main contributions of this work are as follows:
The rest of the paper is structured as follows: In Section 2, we briefly describe the generalized expectation criteria and present the proposed regularization terms  In Section 3, we conduct extensive experiments to justify the proposed methods  We survey related work in Section 4, and summarize our work in Section 5 
Method
We address the robustness problem on top of GE-FL BIBREF0 , a GE method which leverages labeled features as prior knowledge  A labeled feature is a strong indicator of a specific class and is manually provided to the classifier  For example, words like amazing, exciting can be labeled features for class positive in sentiment classification 
Generalized Expectation Criteria
Generalized expectation (GE) criteria BIBREF7 provides us a natural way to directly constrain the model in the preferred direction  For example, when we know the proportion of each class of the dataset in a classification task, we can guide the model to predict out a pre-specified class distribution 
Formally, in a parameter estimation objective function, a GE term expresses preferences on the value of some constraint functions about the model's expectation  Given a constraint function $G({\rm x}, y)$ , a conditional model distribution $p_\theta (y|\rm x)$ , an empirical distribution $\tilde{p}({\rm x})$ over input samples and a score function $S$ , a GE term can be expressed as follows:
$$S(E_{\tilde{p}({\rm x})}[E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]])$$   (Eq  4)
Learning from Labeled Features
Druck et al  ge-fl proposed GE-FL to learn from labeled features using generalized expectation criteria  When given a set of labeled features $K$ , the reference distribution over classes of these features is denoted by $\hat{p}(y| x_k), k \in K$   GE-FL introduces the divergence between this reference distribution and the model predicted distribution $p_\theta (y | x_k)$ , as a term of the objective function:
$$\mathcal {O} = \sum _{k \in K} KL(\hat{p}(y|x_k) || p_\theta (y | x_k)) + \sum _{y,i} \frac{\theta _{yi}^2}{2 \sigma ^2}$$   (Eq  6)
where $\theta _{yi}$ is the model parameter which indicates the importance of word $i$ to class $y$   The predicted distribution $p_\theta (y | x_k)$ can be expressed as follows: $ p_\theta (y | x_k) = \frac{1}{C_k} \sum _{\rm x} p_\theta (y|{\rm x})I(x_k) $
in which $I(x_k)$ is 1 if feature $k$ occurs in instance ${\rm x}$ and 0 otherwise, $C_k = \sum _{\rm x} I(x_k)$ is the number of instances with a non-zero value of feature $k$ , and $p_\theta (y|{\rm x})$ takes a softmax form as follows: $ p_\theta (y|{\rm x}) = \frac{1}{Z(\rm x)}\exp (\sum _i \theta _{yi}x_i)  $
To solve the optimization problem, L-BFGS can be used for parameter estimation 
In the framework of GE, this term can be obtained by setting the constraint function $G({\rm x}, y) = \frac{1}{C_k} \vec{I} (y)I(x_k)$ , where $\vec{I}(y)$ is an indicator vector with 1 at the index corresponding to label $y$ and 0 elsewhere 
Regularization Terms
GE-FL reduces the heavy load of instance annotation and performs well when we provide prior knowledge with no bias  In our experiments, we observe that comparable numbers of labeled features for each class have to be supplied  But as mentioned before, it is often the case that we are not able to provide enough knowledge for some of the classes  For the baseball-hockey classification task, as shown before, GE-FL will predict most of the instances as baseball  In this section, we will show three terms to make the model more robust 
Neutral features are features that are not informative indicator of any classes, for instance, word player to the baseball-hockey classification task  Such features are usually frequent words across all categories  When we set the preference distribution of the neutral features to be uniform distributed, these neutral features will prevent the model from biasing to the class that has a dominate number of labeled features 
Formally, given a set of neutral features $K^{^{\prime }}$ , the uniform distribution is $\hat{p}_u(y|x_k) = \frac{1}{|C|}, k \in K^{^{\prime }}$ , where $|C|$ is the number of classes  The objective function with the new term becomes
$$\mathcal {O}_{NE} = \mathcal {O} + \sum _{k \in K^{^{\prime }}} KL(\hat{p}_u(y|x_k) || p_\theta (y | x_k)) $$   (Eq  9)
Note that we do not need manual annotation to provide neutral features  One simple way is to take the most common features as neutral features  Experimental results show that this strategy works successfully 
Another way to prevent the model from drifting from the desired direction is to constrain the predicted class distribution on unlabeled data  When lacking knowledge about the class distribution of the data, one feasible way is to take maximum entropy principle, as below:
$$\mathcal {O}_{ME} = \mathcal {O} + \lambda \sum _{y} p(y) \log p(y)$$   (Eq  11)
where $p(y)$ is the predicted class distribution, given by $ p(y) = \frac{1}{|X|} \sum _{\rm x} p_\theta (y | \rm x)  $ To control the influence of this term on the overall objective function, we can tune $\lambda $ according to the difference in the number of labeled features of each class  In this paper, we simply set $\lambda $ to be proportional to the total number of labeled features, say $\lambda = \beta |K|$  
This maximum entropy term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$   Therefore, $E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]$ is just the model distribution $p_\theta (y|{\rm x})$ and its expectation with the empirical distribution $\tilde{p}(\rm x)$ is simply the average over input samples, namely $p(y)$   When $S$ takes the maximum entropy form, we can derive the objective function as above 
Sometimes, we have already had much knowledge about the corpus, and can estimate the class distribution roughly without labeling instances  Therefore, we introduce the KL divergence between the predicted and reference class distributions into the objective function 
Given the preference class distribution $\hat{p}(y)$ , we modify the objective function as follows:
$$\mathcal {O}_{KL} &= \mathcal {O} + \lambda KL(\hat{p}(y) || p(y))$$   (Eq  13)
Similarly, we set $\lambda = \beta |K|$  
This divergence term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$ and setting the score function to $S(\hat{p}, p) = \sum _i \hat{p}_i \log \frac{\hat{p}_i}{p_i}$ , where $p$ and $\hat{p}$ are distributions  Note that this regularization term involves the reference class distribution which will be discussed later 
Experiments
In this section, we first justify the approach when there exists unbalance in the number of labeled features or in class distribution  Then, to test the influence of $\lambda $ , we conduct some experiments with the method which incorporates the KL divergence of class distribution  Last, we evaluate our approaches in 9 commonly used text classification datasets  We set $\lambda = 5|K|$ by default in all experiments unless there is explicit declaration  The baseline we choose here is GE-FL BIBREF0 , a method based on generalization expectation criteria 
Data Preparation
We evaluate our methods on several commonly used datasets whose themes range from sentiment, web-page, science to medical and healthcare  We use bag-of-words feature and remove stopwords in the preprocess stage  Though we have labels of all documents, we do not use them during the learning process, instead, we use the label of features 
The movie dataset, in which the task is to classify the movie reviews as positive or negtive, is used for testing the proposed approaches with unbalanced labeled features, unbalanced datasets or different $\lambda $ parameters  All unbalanced datasets are constructed based on the movie dataset by randomly removing documents of the positive class  For each experiment, we conduct 10-fold cross validation 
As described in BIBREF0 , there are two ways to obtain labeled features  The first way is to use information gain  We first calculate the mutual information of all features according to the labels of the documents and select the top 20 as labeled features for each class as a feature pool  Note that using information gain requires the document label, but this is only to simulate how we human provide prior knowledge to the model  The second way is to use LDA BIBREF9 to select features  We use the same selection process as BIBREF0 , where they first train a LDA on the dataset, and then select the most probable features of each topic (sorted by $P(w_i|t_j)$ , the probability of word $w_i$ given topic $t_j$ ) 
Similar to BIBREF10 , BIBREF0 , we estimate the reference distribution of the labeled features using a heuristic strategy  If there are $|C|$ classes in total, and $n$ classes are associated with a feature $k$ , the probability that feature $k$ is related with any one of the $n$ classes is $\frac{0 9}{n}$ and with any other class is $\frac{0 1}{|C| - n}$  
Neutral features are the most frequent words after removing stop words, and their reference distributions are uniformly distributed  We use the top 10 frequent words as neutral features in all experiments 
With Unbalanced Labeled Features
In this section, we evaluate our approach when there is unbalanced knowledge on the categories to be classified  The labeled features are obtained through information gain  Two settings are chosen:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other  The original balanced movie dataset is used (positive:negative=1:1) 
(b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4) 
As shown in Figure 1 , Maximum entropy principle shows improvement only on the balanced case

ground_truth: Unanswerable

Answer with content missing: (Parent subsections) combine precisions for n-gram orders 1-4

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. which languages are explored?
Context 4:  An obvious reason is that maximum entropy only favors uniform distribution 
Incorporating Neutral features performs similarly to maximum entropy since we assume that neutral words are uniformly distributed  Its accuracy decreases slowly when the number of labeled features becomes larger ( $t>4$ ) (Figure 1 (a)), suggesting that the model gradually biases to the class with more labeled features, just like GE-FL 
Incorporating the KL divergence of class distribution performs much better than GE-FL on both balanced and unbalanced datasets  This shows that it is effective to control the unbalance in labeled features and in the dataset 
With Balanced Labeled Features
We also compare with the baseline when the labeled features are balanced  Similar to the experiment above, the labeled features are obtained by information gain  Two settings are experimented with:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for each class, and conduct comparisons on the original balanced movie dataset (positive:negtive=1:1) 
(b) Similar to (a), but the class distribution is unbalanced, by randomly removing 75% positive documents (positive:negative=1:4) 
Results are shown in Figure 2   When the dataset is balanced (Figure 2 (a)), there is little difference between GE-FL and our methods  The reason is that the proposed regularization terms provide no additional knowledge to the model and there is no bias in the labeled features  On the unbalanced dataset (Figure 2 (b)), incorporating KL divergence is much better than GE-FL since we provide additional knowledge(the true class distribution), but maximum entropy and neutral features are much worse because forcing the model to approach the uniform distribution misleads it 
With Unbalanced Class Distributions
Our methods are also evaluated on datasets with different unbalanced class distributions  We manually construct several movie datasets with class distributions of 1:2, 1:3, 1:4 by randomly removing 50%, 67%, 75% positive documents  The original balanced movie dataset is used as a control group  We test with both balanced and unbalanced labeled features  For the balanced case, we randomly select 10 features from the feature pool for each class, and for the unbalanced case, we select 10 features for one class, and 1 feature for the other  Results are shown in Figure 3  
Figure 3 (a) shows that when the dataset and the labeled features are both balanced, there is little difference between our methods and GE-FL(also see Figure 2 (a))  But when the class distribution becomes more unbalanced, the difference becomes more remarkable  Performance of neutral features and maximum entropy decrease significantly but incorporating KL divergence increases remarkably  This suggests if we have more accurate knowledge about class distribution, KL divergence can guide the model to the right direction 
Figure 3 (b) shows that when the labeled features are unbalanced, our methods significantly outperforms GE-FL  Incorporating KL divergence is robust enough to control unbalance both in the dataset and in labeled features while the other three methods are not so competitive 
The Influence of λ\lambda 
We present the influence of $\lambda $ on the method that incorporates KL divergence in this section  Since we simply set $\lambda = \beta |K|$ , we just tune $\beta $ here  Note that when $\beta = 0$ , the newly introduced regularization term is disappeared, and thus the model is actually GE-FL  Again, we test the method with different $\lambda $ in two settings:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other class  The original balanced movie dataset is used (positive:negative=1:1) 
(b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4) 
Results are shown in Figure 4   As expected, $\lambda $ reflects how strong the regularization is  The model tends to be closer to our preferences with the increasing of $\lambda $ on both cases 
Using LDA Selected Features
We compare our methods with GE-FL on all the 9 datasets in this section  Instead of using features obtained by information gain, we use LDA to select labeled features  Unlike information gain, LDA does not employ any instance labels to find labeled features  In this setting, we can build classification models without any instance annotation, but just with labeled features 
Table 1 shows that our three methods significantly outperform GE-FL  Incorporating neutral features performs better than GE-FL on 7 of the 9 datasets, maximum entropy is better on 8 datasets, and KL divergence better on 7 datasets 
LDA selects out the most predictive features as labeled features without considering the balance among classes  GE-FL does not exert any control on such an issue, so the performance is severely suffered  Our methods introduce auxiliary regularization terms to control such a bias problem and thus promote the model significantly 
Related Work
There have been much work that incorporate prior knowledge into learning, and two related lines are surveyed here  One is to use prior knowledge to label unlabeled instances and then apply a standard learning algorithm  The other is to constrain the model directly with prior knowledge 
Liu et al text manually labeled features which are highly predictive to unsupervised clustering assignments and use them to label unlabeled data  Chang et al guiding proposed constraint driven learning  They first used constraints and the learned model to annotate unlabeled instances, and then updated the model with the newly labeled data  Daumé daume2008cross proposed a self training method in which several models are trained on the same dataset, and only unlabeled instances that satisfy the cross task knowledge constraints are used in the self training process 
MaCallum et al gec proposed generalized expectation(GE) criteria which formalised the knowledge as constraint terms about the expectation of the model into the objective function Graça et al pr proposed posterior regularization(PR) framework which projects the model's posterior onto a set of distributions that satisfy the auxiliary constraints  Druck et al ge-fl explored constraints of labeled features in the framework of GE by forcing the model's predicted feature distribution to approach the reference distribution  Andrzejewski et al andrzejewski2011framework proposed a framework in which general domain knowledge can be easily incorporated into LDA  Altendorf et al altendorf2012learning explored monotonicity constraints to improve the accuracy while learning from sparse data  Chen et al chen2013leveraging tried to learn comprehensible topic models by leveraging multi-domain knowledge 
Mann and McCallum simple,generalized incorporated not only labeled features but also other knowledge like class distribution into the objective function of GE-FL  But they discussed only from the semi-supervised perspective and did not investigate into the robustness problem, unlike what we addressed in this paper 
There are also some active learning methods trying to use prior knowledge  Raghavan et al feedback proposed to use feedback on instances and features interlacedly, and demonstrated that feedback on features boosts the model much  Druck et al active proposed an active learning method which solicits labels on features rather than on instances and then used GE-FL to train the model 
Conclusion and Discussions
This paper investigates into the problem of how to leverage prior knowledge robustly in learning models  We propose three regularization terms on top of generalized expectation criteria  As demonstrated by the experimental results, the performance can be considerably improved when taking into account these factors  Comparative results show that our proposed methods is more effective and works more robustly against baselines  To the best of our knowledge, this is the first work to address the robustness problem of leveraging knowledge, and may inspire other research 
We then present more detailed discussions about the three regularization methods  Incorporating neutral features is the simplest way of regularization, which doesn't require any modification of GE-FL but just finding out some common features  But as Figure 1 (a) shows, only using neutral features are not strong enough to handle extremely unbalanced labeled features 
The maximum entropy regularization term shows the strong ability of controlling unbalance 
This method doesn't need any extra knowledge, and is thus suitable when we know nothing about the corpus  But this method assumes that the categories are uniformly distributed, which may not be the case in practice, and it will have a degraded performance if the assumption is violated (see Figure 1 (b), Figure 2 (b), Figure 3 (a)) 
The KL divergence performs much better on unbalanced corpora than other methods  The reason is that KL divergence utilizes the reference class distribution and doesn't make any assumptions  The fact suggests that additional knowledge does benefit the model 
However, the KL divergence term requires providing the true class distribution  Sometimes, we may have the exact knowledge about the true distribution, but sometimes we may not  Fortunately, the model is insensitive to the true distribution and therefore a rough estimation of the true distribution is sufficient  In our experiments, when the true class distribution is 1:2, where the reference class distribution is set to 1:1 5/1:2/1:2 5, the accuracy is 0 755/0 756/0 760 respectively  This provides us the possibility to perform simple computing on the corpus to obtain the distribution in reality  Or, we can set the distribution roughly with domain expertise 

ground_truth: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish

Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish , Swedish