prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What does an individual model consist of?
Context 1: Introduction
Semantic Role Labeling (SRL) has emerged as an important task in Natural Language Processing (NLP) due to its applicability in information extraction, question answering, and other NLP tasks  SRL is the problem of finding predicate-argument structure in a sentence, as illustrated below:
INLINEFORM0
Here, the predicate WRITE has two arguments: `Mike' as A0 or the writer, and `a book' as A1 or the thing written  The labels A0 and A1 correspond to the PropBank annotations BIBREF0  
As the need for SRL arises in different domains and languages, the existing manually annotated corpora become insufficient to build supervised systems  This has motivated work on unsupervised SRL BIBREF1 , BIBREF2 , BIBREF3   Previous work has indicated that unsupervised systems could benefit from the word alignment information in parallel text in two or more languages BIBREF4 , BIBREF5 , BIBREF6   For example, consider the German translation of sentence INLINEFORM0 :
INLINEFORM0
If sentences INLINEFORM0 and INLINEFORM1 have the word alignments: Mike-Mike, written-geschrieben, and book-Buch, the system might be able to predict A1 for Buch, even if there is insufficient information in the monolingual German data to learn this assignment  Thus, in languages where the resources are sparse or not good enough, or the distributions are not informative, SRL systems could be made more accurate by using parallel data with resource rich or more amenable languages 
In this paper, we propose a joint Bayesian model for unsupervised semantic role induction in multiple languages  The model consists of individual Bayesian models for each language BIBREF3 , and crosslingual latent variables to incorporate soft role agreement between aligned constituents  This latent variable approach has been demonstrated to increase the performance in a multilingual unsupervised part-of-speech tagging model based on HMMs BIBREF4   We investigate the application of this approach to unsupervised SRL, presenting the performance improvements obtained in different settings involving labeled and unlabeled data, and analyzing the annotation effort required to obtain similar gains using labeled data 
We begin by briefly describing the unsupervised SRL pipeline and the monolingual semantic role induction model we use, and then describe our multilingual model 
Unsupervised SRL Pipeline
As established in previous work BIBREF7 , BIBREF8 , we use a standard unsupervised SRL setup, consisting of the following steps:
The task we model, unsupervised semantic role induction, is the step 4 of this pipeline 
Monolingual Model
We use the Bayesian model of garg2012unsupervised as our base monolingual model  The semantic roles are predicate-specific  To model the role ordering and repetition preferences, the role inventory for each predicate is divided into Primary and Secondary roles as follows:
For example, the complete role sequence in a frame could be: INLINEFORM0 INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 , INLINEFORM8 INLINEFORM9   The ordering is defined as the sequence of PRs, INLINEFORM10 INLINEFORM11 , INLINEFORM12 , INLINEFORM13 , INLINEFORM14 , INLINEFORM15 INLINEFORM16   Each pair of consecutive PRs in an ordering is called an interval  Thus, INLINEFORM17 is an interval that contains two SRs, INLINEFORM18 and INLINEFORM19   An interval could also be empty, for instance INLINEFORM20 contains no SRs  When we evaluate, these roles get mapped to gold roles  For instance, the PR INLINEFORM21 could get mapped to a core role like INLINEFORM22 , INLINEFORM23 , etc  or to a modifier role like INLINEFORM24 , INLINEFORM25 , etc  garg2012unsupervised reported that, in practice, PRs mostly get mapped to core roles and SRs to modifier roles, which conforms to the linguistic motivations for this distinction 
Figure FIGREF16 illustrates two copies of the monolingual model, on either side of the crosslingual latent variables  The generative process is as follows:
All the multinomial and binomial distributions have symmetric Dirichlet and beta priors respectively  Figure FIGREF7 gives the probability equations for the monolingual model  This formulation models the global role ordering and repetition preferences using PRs, and limited context for SRs using intervals  Ordering and repetition information was found to be helpful in supervised SRL as well BIBREF9 , BIBREF8 , BIBREF10   More details, including the motivations behind this model, are in BIBREF3  
Multilingual Model
The multilingual model uses word alignments between sentences in a parallel corpus to exploit role correspondences across languages  We make copies of the monolingual model for each language and add additional crosslingual latent variables (CLVs) to couple the monolingual models, capturing crosslingual semantic role patterns  Concretely, when training on parallel sentences, whenever the head words of the arguments are aligned, we add a CLV as a parent of the two corresponding role variables  Figure FIGREF16 illustrates this model  The generative process, as explained below, remains the same as the monolingual model for the most part, with the exception of aligned roles which are now generated by both the monolingual process as well as the CLV 
Every predicate-tuple has its own inventory of CLVs specific to that tuple  Each CLV INLINEFORM0 is a multi-valued variable where each value defines a distribution over role labels for each language (denoted by INLINEFORM1 above)  These distributions over labels are trained to be peaky, so that each value INLINEFORM2 for a CLV represents a correlation between the labels that INLINEFORM3 predicts in the two languages  For example, a value INLINEFORM4 for the CLV INLINEFORM5 might give high probabilities to INLINEFORM6 and INLINEFORM7 in language 1, and to INLINEFORM8 in language 2  If INLINEFORM9 is the only value for INLINEFORM10 that gives high probability to INLINEFORM11 in language 1, and the monolingual model in language 1 decides to assign INLINEFORM12 to the role for INLINEFORM13 , then INLINEFORM14 will predict INLINEFORM15 in language 2, with high probability  We generate the CLVs via a Chinese Restaurant Process BIBREF11 , a non-parametric Bayesian model, which allows us to induce the number of CLVs for every predicate-tuple from the data  We continue to train on the non-parallel sentences using the respective monolingual models 
The multilingual model is deficient, since the aligned roles are being generated twice  Ideally, we would like to add the CLV as additional conditioning variables in the monolingual models  The new joint probability can be written as equation UID11 (Figure FIGREF7 ), which can be further decomposed following the decomposition of the monolingual model in Figure FIGREF7   However, having this additional conditioning variable breaks the Dirichlet-multinomial conjugacy, which makes it intractable to marginalize out the parameters during inference  Hence, we use an approximation where we treat each of the aligned roles as being generated twice, once by the monolingual model and once by the corresponding CLV (equation ) 
This is the first work to incorporate the coupling of aligned arguments directly in a Bayesian SRL model  This makes it easier to see how to extend this model in a principled way to incorporate additional sources of information  First, the model scales gracefully to more than two languages  If there are a total of INLINEFORM0 languages, and there is an aligned argument in INLINEFORM1 of them, the multilingual latent variable is connected to only those INLINEFORM2 aligned arguments 
Second, having one joint Bayesian model allows us to use the same model in various semi-supervised learning settings, just by fixing the annotated variables during training  Section SECREF29 evaluates a setting where we have some labeled data in one language (called source), while no labeled data in the second language (called target)  Note that this is different from a classic annotation projection setting (e g  BIBREF12 ), where the role labels are mapped from source constituents to aligned target constituents 
Inference and Training
The inference problem consists of predicting the role labels and CLVs (the hidden variables) given the predicate, its voice, and syntactic features of all the identified arguments (the visible variables)  We use a collapsed Gibbs-sampling based approach to generate samples for the hidden variables (model parameters are integrated out)  The sample counts and the priors are then used to calculate the MAP estimate of the model parameters 
For the monolingual model, the role at a given position is sampled as:
DISPLAYFORM0
where the subscript INLINEFORM0 refers to all the variables except at position INLINEFORM1 , INLINEFORM2 refers to the variables in all the training instances except the current one, and INLINEFORM3 refers to all the model parameters  The above integral has a closed form solution due to Dirichlet-multinomial conjugacy 
For sampling roles in the multilingual model, we also need to consider the probabilities of roles being generated by the CLVs:
DISPLAYFORM0
For sampling CLVs, we need to consider three factors: two corresponding to probabilities of generating the aligned roles, and the third one corresponding to selecting the CLV according to CRP 
DISPLAYFORM0
where the aligned roles INLINEFORM0 and INLINEFORM1 are connected to INLINEFORM2 , and INLINEFORM3 refers to all the variables except INLINEFORM4 , INLINEFORM5 , and INLINEFORM6  
We use the trained parameters to parse the monolingual data using the monolingual model  The crosslingual parameters are ignored even if they were used during training  Thus, the information coming from the CLVs acts as a regularizer for the monolingual models 
Evaluation
Following the setting of titovcrosslingual, we evaluate only on the arguments that were correctly identified, as the incorrectly identified arguments do not have any gold semantic labels  Evaluation is done using the metric proposed by lang2011unsupervised, which has 3 components: (i) Purity (PU) measures how well an induced cluster corresponds to a single gold role, (ii) Collocation (CO) measures how well a gold role corresponds to a single induced cluster, and (iii) F1 is the harmonic mean of PU and CO  For each predicate, let INLINEFORM0 denote the total number of argument instances, INLINEFORM1 the instances in the induced cluster INLINEFORM2 , and INLINEFORM3 the instances having label INLINEFORM4 in gold annotations  INLINEFORM5 , INLINEFORM6 , and INLINEFORM7   The score for each predicate is weighted by the number of its argument instances, and a weighted average is computed over all the predicates 
Baseline
We use the same baseline as used by lang2011unsupervised which has been shown to be difficult to outperform  This baseline assigns a semantic role to a constituent based on its syntactic function, i e  the dependency relation to its head  If there is a total of INLINEFORM0 clusters, INLINEFORM1 most frequent syntactic functions get a cluster each, and the rest are assigned to the INLINEFORM2 th cluster 
Closest Previous Work
This work is closely related to the cross-lingual unsupervised SRL work of titovcrosslingual  Their model has separate monolingual models for each language and an extra penalty term which tries to maximize INLINEFORM0 and INLINEFORM1 i e  for all the aligned arguments with role label INLINEFORM2 in language 1, it tries to find a role label INLINEFORM3 in language 2 such that the given proportion is maximized and vice verse  However, there is no efficient way to optimize the objective with this penalty term and the authors used an inference method similar to annotation projection  Further, the method does not scale naturally to more than two languages  Their algorithm first does monolingual inference in one language ignoring the penalty and then does the inference in the second language taking into account the penalty term  In contrast, our model adds the latent variables as a part of the model itself, and not an external penalty, which enables us to use the standard Bayesian learning methods such as sampling 
The monolingual model we use BIBREF3 also has two main advantages over titovcrosslingual  First, the former incorporates a global role ordering probability that is missing in the latter  Secondly, the latter defines argument-keys as a tuple of four syntactic features and all the arguments having the same argument-keys are assigned the same role  This kind of hard clustering is avoided in the former model where two constituents having the same set of features might get assigned different roles if they appear in different contexts 
Data
Following titovcrosslingual, we run our experiments on the English (EN) and German (DE) sections of the CoNLL 2009 corpus BIBREF13 , and EN-DE section of the Europarl corpus BIBREF14   We get about 40k EN and 36k DE sentences from the CoNLL 2009 training set, and about 1 5M parallel EN-DE sentences from Europarl  For appropriate comparison, we keep the same setting as in BIBREF6 for automatic parses and argument identification, which we briefly describe here  The EN sentences are parsed syntactically using MaltParser BIBREF15 and DE using LTH parser BIBREF16   All the non-auxiliary verbs are selected as predicates  In CoNLL data, this gives us about 3k EN and 500 DE predicates  The total number of predicate instances are 3 4M in EN (89k CoNLL + 3 3M Europarl) and 2 62M in DE (17k CoNLL + 2 6M Europarl)  The arguments for EN are identified using the heuristics proposed by lang2011unsupervised  However, we get an F1 score of 85 1% for argument identification on CoNLL 2009 EN data as opposed to 80 7% reported by titovcrosslingual  This could be due to implementation differences, which unfortunately makes our EN results incomparable  For DE, the arguments are identified using the LTH system BIBREF16 , which gives an F1 score of 86 5% on the CoNLL 2009 DE data  The word alignments for the EN-DE parallel Europarl corpus are computed using GIZA++ BIBREF17   For high-precision, only the intersecting alignments in the two directions are kept  We define two semantic arguments as aligned if their head-words are aligned  In total we get 9 3M arguments for EN (240k CoNLL + 9 1M Europarl) and 4 43M for DE (32k CoNLL + 4 4M Europarl)  Out of these, 0 76M arguments are aligned 
Main Results
Since the CoNLL annotations have 21 semantic roles in total, we use 21 roles in our model as well as the baseline  Following garg2012unsupervised, we set the number of PRs to 2 (excluding INLINEFORM0 , INLINEFORM1 and INLINEFORM2 ), and SRs to 21-2=19  Table TABREF27 shows the results 
In the first setting (Line 1), we train and test the monolingual model on the CoNLL data  We observe significant improvements in F1 score over the Baseline (Line 0) in both languages  Using the CoNLL 2009 dataset alone, titovcrosslingual report an F1 score of 80 9% (PU=86 8%, CO=75 7%) for German  Thus, our monolingual model outperforms their monolingual model in German  For English, they report an F1 score of 83 6% (PU=87 5%, CO=80 1%), but note that our English results are not directly comparable to theirs due to differences argument identification, as discussed in section SECREF25   As their argument identification score is lower, perhaps their system is discarding “difficult” arguments which leads to a higher clustering score 
In the second setting (Line 2), we use the additional monolingual Europarl (EP) data for training  We get equivalent results in English and a significant improvement in German compared to our previous setting (Line 1)  The German dataset in CoNLL is quite small and benefits from the additional EP training data  In contrast, the English model is already quite good due to a relatively big dataset from CoNLL, and good accuracy syntactic parsers  Unfortunately, titovcrosslingual do not report results with this setting 
The third setting (Line 3) gives the results of our multilingual model, which adds the word alignments in the EP data  Comparing with Line 2, we get non-significant improvements in both languages  titovcrosslingual obtain an F1 score of 82 7% (PU=85 0%, CO=80 6%) for German, and 83 7% (PU=86 8%, CO=80 7%) for English  Thus, for German, our multilingual Bayesian model is able to capture the cross-lingual patterns at least as well as the external penalty term in BIBREF6   We cannot compare the English results unfortunately due to differences in argument identification 
We also compared monolingual and bilingual training data using a setting that emulates the standard supervised setup of separate training and test data sets  We train only on the EP dataset and test on the CoNLL dataset  Lines 4 and 5 of Table TABREF27 give the results  The multilingual model obtains small improvements in both languages, which confirms the results from the standard unsupervised setup, comparing lines 2 to 3 
These results indicate that little information can be learned about semantic roles from this parallel data setup  One possible explanation for this result is that the setup itself is inadequate  Given the definition of aligned arguments, only 8% of English arguments and 17% of German arguments are aligned  This plus our experiments suggest that improving the alignment model is a necessary step to making effective use of parallel data in multilingual SRI, for example by joint modeling with SRI  We leave this exploration to future work 
Multilingual Training with Labeled Data for One Language
Another motivation for jointly modeling SRL in multiple languages is the transfer of information from a resource rich language to a resource poor language  We evaluated our model in a very general annotation transfer scenario, where we have a small labeled dataset for one language (source), and a large parallel unlabeled dataset for the source and another (target) language  We investigate whether this setting improves the parameter estimates for the target language  To this end, we clamp the role annotations of the source language in the CoNLL dataset using a predefined mapping, and do not sample them during training  This data gives us good parameters for the source language, which are used to sample the roles of the source language in the unlabeled Europarl data  The CLVs aim to capture this improvement and thereby improve sampling and parameter estimates for the target language  Table TABREF28 shows the results of this experiment  We obtain small improvements in the target languages  As in the unsupervised setting, the small percentage of aligned roles probably limits the impact of the cross-lingual information 
Labeled Data in Monolingual Model
We explored the improvement in the monolingual model in a semi-supervised setting  To this end, we randomly selected INLINEFORM0 of the sentences in the CoNLL dataset as “supervised sentences” and the rest INLINEFORM1 were kept unsupervised  Next, we clamped the role labels of the supervised sentences using the predefined mapping from Section SECREF29   Sampling was done on the unsupervised sentences as usual  We then measured the clustering performance using the trained parameters 
To access the contribution of partial supervision better, we constructed a “supervised baseline” as follows  For predicates seen in the supervised sentences, a MAP estimate of the parameters was calculated using the predefined mapping  For the unseen predicates, the standard baseline was used 
Figures FIGREF33 and FIGREF33 show the performance variation with INLINEFORM0   We make the following observations:
[leftmargin=*]
In both languages, at around INLINEFORM0 , the supervised baseline starts outperforming the semi-supervised model, which suggests that manually labeling about 10% of the sentences is a good enough alternative to our training procedure  Note that 10% amounts to about 3 6k sentences in German and 4k in English  We noticed that the proportion of seen predicates increases dramatically as we increase the proportion of supervised sentences  At 10% supervised sentences, the model has already seen 63% of predicates in German and 44% in English  This explains to some extent why only 10% labeled sentences are enough 
For German, it takes about 3 5% or 1260 supervised sentences to have the same performance increase as 1 5M unlabeled sentences (Line 1 to Line 2 in Table TABREF27 )  Adding about 180 more supervised sentences also covers the benefit obtained by alignments in the multilingual model (Line 2 to Line 3 in Table TABREF27 )  There is no noticeable performance difference in English 
We also evaluated the performance variation on a completely unseen CoNLL test set  Since the test set is very small compared to the training set, the clustering evaluation is not as reliable  Nonetheless, we broadly obtained the same pattern 
Related Work
As discussed in section SECREF24 , our work is closely related to the crosslingual unsupervised SRL work of titovcrosslingual  The idea of using superlingual latent variables to capture cross-lingual information was proposed for POS tagging by naseem2009multilingual, which we use here for SRL  In a semi-supervised setting, pado2009cross used a graph based approach to transfer semantic role annotations from English to German  furstenau2009graph used a graph alignment method to measure the semantic and syntactic similarity between dependency tree arguments of known and unknown verbs 
For monolingual unsupervised SRL, swier2004unsupervised presented the first work on a domain-general corpus, the British National Corpus, using 54 verbs taken from VerbNet  garg2012unsupervised proposed a Bayesian model for this problem that we use here  titov2012bayesian also proposed a closely related Bayesian model  grenager2006unsupervised proposed a generative model but their parameter space consisted of all possible linkings of syntactic constituents and semantic roles, which made unsupervised learning difficult and a separate language-specific rule based method had to be used to constrain this space  Other proposed models include an iterative split-merge algorithm BIBREF18 and a graph-partitioning based approach BIBREF1   marquez2008semantic provide a good overview of the supervised SRL systems 
Conclusions
We propose a Bayesian model of semantic role induction (SRI) that uses crosslingual latent variables to capture role alignments in parallel corpora  The crosslingual latent variables capture correlations between roles in different languages, and regularize the parameter estimates of the monolingual models  Because this is a joint Bayesian model of multilingual SRI, we can apply the same model to a variety of training scenarios just by changing the inference procedure appropriately  We evaluate monolingual SRI with a large unlabeled dataset, bilingual SRI with a parallel corpus, bilingual SRI with annotations available for the source language, and monolingual SRI with a small labeled dataset  Increasing the amount of monolingual unlabeled data significantly improves SRI in German but not in English  Adding word alignments in parallel sentences results in small, non significant improvements, even if there is some labeled data available in the source language  This difficulty in showing the usefulness of parallel corpora for SRI may be due to the current assumptions about role alignments, which mean that only a small percentage of roles are aligned  Further analyses reveals that annotating small amounts of data can easily outperform the performance gains obtained by adding large unlabeled dataset as well as adding parallel corpora 
Future work includes training on different language pairs, on more than two languages, and with more inclusive models of role alignment 
Acknowledgments
This work was funded by the Swiss NSF grant 200021_125137 and EC FP7 grant PARLANCE 
Context 2: Introduction
Named Entity Recognition (NER) is a foremost NLP task to label each atomic elements of a sentence into specific categories like "PERSON", "LOCATION", "ORGANIZATION" and othersBIBREF0  There has been an extensive NER research on English, German, Dutch and Spanish language BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, and notable research on low resource South Asian languages like HindiBIBREF6, IndonesianBIBREF7 and other Indian languages (Kannada, Malayalam, Tamil and Telugu)BIBREF8  However, there has been no study on developing neural NER for Nepali language  In this paper, we propose a neural based Nepali NER using latest state-of-the-art architecture based on grapheme-level which doesn't require any hand-crafted features and no data pre-processing 
Recent neural architecture like BIBREF1 is used to relax the need to hand-craft the features and need to use part-of-speech tag to determine the category of the entity  However, this architecture have been studied for languages like English, and German and not been applied to languages like Nepali which is a low resource language i e limited data set to train the model  Traditional methods like Hidden Markov Model (HMM) with rule based approachesBIBREF9,BIBREF10, and Support Vector Machine (SVM) with manual feature-engineeringBIBREF11 have been applied but they perform poor compared to neural  However, there has been no research in Nepali NER using neural network  Therefore, we created the named entity annotated dataset partly with the help of Dataturk to train a neural model  The texts used for this dataset are collected from various daily news sources from Nepal around the year 2015-2016 
Following are our contributions:
We present a novel Named Entity Recognizer (NER) for Nepali language  To best of our knowledge we are the first to propose neural based Nepali NER 
As there are not good quality dataset to train NER we release a dataset to support future research
We perform empirical evaluation of our model with state-of-the-art models with relative improvement of upto 10%
In this paper, we present works similar to ours in Section SECREF2  We describe our approach and dataset statistics in Section SECREF3 and SECREF4, followed by our experiments, evaluation and discussion in Section SECREF5, SECREF6, and SECREF7  We conclude with our observations in Section SECREF8 
To facilitate further research our code and dataset will be made available at github com/link-yet-to-be-updated
Related Work
There has been a handful of research on Nepali NER task based on approaches like Support Vector Machine and gazetteer listBIBREF11 and Hidden Markov Model and gazetteer listBIBREF9,BIBREF10 
BIBREF11 uses SVM along with features like first word, word length, digit features and gazetteer (person, organization, location, middle name, verb, designation and others)  It uses one vs rest classification model to classify each word into different entity classes  However, it does not the take context word into account while training the model  Similarly, BIBREF9 and BIBREF10 uses Hidden Markov Model with n-gram technique for extracting POS-tags  POS-tags with common noun, proper noun or combination of both are combined together, then uses gazetteer list as look-up table to identify the named entities 
Researchers have shown that the neural networks like CNNBIBREF12, RNNBIBREF13, LSTMBIBREF14, GRUBIBREF15 can capture the semantic knowledge of language better with the help of pre-trained embbeddings like word2vecBIBREF16, gloveBIBREF17 or fasttextBIBREF18 
Similar approaches has been applied to many South Asian languages like HindiBIBREF6, IndonesianBIBREF7, BengaliBIBREF19 and In this paper, we present the neural network architecture for NER task in Nepali language, which doesn't require any manual feature engineering nor any data pre-processing during training  First we are comparing BiLSTMBIBREF14, BiLSTM+CNNBIBREF20, BiLSTM+CRFBIBREF1, BiLSTM+CNN+CRFBIBREF2 models with CNN modelBIBREF0 and Stanford CRF modelBIBREF21  Secondly, we show the comparison between models trained on general word embeddings, word embedding + character-level embedding, word embedding + part-of-speech(POS) one-hot encoding and word embedding + grapheme clustered or sub-word embeddingBIBREF22  The experiments were performed on the dataset that we created and on the dataset received from ILPRL lab  Our extensive study shows that augmenting word embedding with character or grapheme-level representation and POS one-hot encoding vector yields better results compared to using general word embedding alone 
Approach
In this section, we describe our approach in building our model  This model is partly inspired from multiple models BIBREF20,BIBREF1, andBIBREF2
Approach ::: Bidirectional LSTM
We used Bi-directional LSTM to capture the word representation in forward as well as reverse direction of a sentence  Generally, LSTMs take inputs from left (past) of the sentence and computes the hidden state  However, it is proven beneficialBIBREF23 to use bi-directional LSTM, where, hidden states are computed based from right (future) of sentence and both of these hidden states are concatenated to produce the final output as $h_t$=[$\overrightarrow{h_t}$;$\overleftarrow{h_t}$], where $\overrightarrow{h_t}$, $\overleftarrow{h_t}$ = hidden state computed in forward and backward direction respectively 
Approach ::: Features ::: Word embeddings
We have used Word2Vec BIBREF16, GloVe BIBREF17 and FastText BIBREF18 word vectors of 300 dimensions  These vectors were trained on the corpus obtained from Nepali National Corpus  This pre-lemmatized corpus consists of 14 million words from books, web-texts and news papers  This corpus was mixed with the texts from the dataset before training CBOW and skip-gram version of word2vec using gensim libraryBIBREF24  This trained model consists of vectors for 72782 unique words 
Light pre-processing was performed on the corpus before training it  For example, invalid characters or characters other than Devanagari were removed but punctuation and numbers were not removed  We set the window context at 10 and the rare words whose count is below 5 are dropped  These word embeddings were not frozen during the training session because fine-tuning word embedding help achieve better performance compared to frozen oneBIBREF20 
We have used fasttext embeddings in particular because of its sub-word representation ability, which is very useful in highly inflectional language as shown in Table TABREF25  We have trained the word embedding in such a way that the sub-word size remains between 1 and 4  We particularly chose this size because in Nepali language a single letter can also be a word, for example e, t, C, r, l, n, u and a single character (grapheme) or sub-word can be formed after mixture of dependent vowel signs with consonant letters for example, C + O + = CO, here three different consonant letters form a single sub-word 
The two-dimensional visualization of an example word npAl is shown in FIGREF14  Principal Component Analysis (PCA) technique was used to generate this visualization which helps use to analyze the nearest neighbor words of a given sample word  84 and 104 nearest neighbors were observed using word2vec and fasttext embedding respectively on the same corpus 
Approach ::: Features ::: Character-level embeddings
BIBREF20 and BIBREF2 successfully presented that the character-level embeddings, extracted using CNN, when combined with word embeddings enhances the NER model performance significantly, as it is able to capture morphological features of a word  Figure FIGREF7 shows the grapheme-level CNN used in our model, where inputs to CNN are graphemes  Character-level CNN is also built in similar fashion, except the inputs are characters  Grapheme or Character -level embeddings are randomly initialized from [0,1] with real values with uniform distribution of dimension 30 
Approach ::: Features ::: Grapheme-level embeddings
Grapheme is atomic meaningful unit in writing system of any languages  Since, Nepali language is highly morphologically inflectional, we compared grapheme-level representation with character-level representation to evaluate its effect  For example, in character-level embedding, each character of a word npAl results into n + + p + A + l has its own embedding  However, in grapheme level, a word npAl is clustered into graphemes, resulting into n + pA + l  Here, each grapheme has its own embedding  This grapheme-level embedding results good scores on par with character-level embedding in highly inflectional languages like Nepali, because graphemes also capture syntactic information similar to characters  We created grapheme clusters using uniseg package which is helpful in unicode text segmentations 
Approach ::: Features ::: Part-of-speech (POS) one hot encoding
We created one-hot encoded vector of POS tags and then concatenated with pre-trained word embeddings before passing it to BiLSTM network  A sample of data is shown in figure FIGREF13 
Dataset Statistics ::: OurNepali dataset
Since, we there was no publicly available standard Nepali NER dataset and did not receive any dataset from the previous researchers, we had to create our own dataset  This dataset contains the sentences collected from daily newspaper of the year 2015-2016  This dataset has three major classes Person (PER), Location (LOC) and Organization (ORG)  Pre-processing was performed on the text before creation of the dataset, for example all punctuations and numbers besides ',', '-', '|' and ' ' were removed  Currently, the dataset is in standard CoNLL-2003 IO formatBIBREF25 
Since, this dataset is not lemmatized originally, we lemmatized only the post-positions like Ek, kO, l, mA, m, my, jF, sg, aEG which are just the few examples among 299 post positions in Nepali language  We obtained these post-positions from sanjaalcorps and added few more to match our dataset  We will be releasing this list in our github repository  We found out that lemmatizing the post-positions boosted the F1 score by almost 10% 
In order to label our dataset with POS-tags, we first created POS annotated dataset of 6946 sentences and 16225 unique words extracted from POS-tagged Nepali National Corpus and trained a BiLSTM model with 95 14% accuracy which was used to create POS-tags for our dataset 
The dataset released in our github repository contains each word in newline with space separated POS-tags and Entity-tags
Context 3: Introduction
Suppose a user wants to write a sentence “I will be 10 minutes late ” Ideally, she would type just a few keywords such as “10 minutes late” and an autocomplete system would be able to infer the intended sentence (Figure FIGREF1)  Existing left-to-right autocomplete systems BIBREF0, BIBREF1 can often be inefficient, as the prefix of a sentence (e g  “I will be”) fails to capture the core meaning of the sentence  Besides the practical goal of building a better autocomplete system, we are interested in exploring the tradeoffs inherent to such communication schemes between the efficiency of typing keywords, accuracy of reconstruction, and interpretability of keywords 
One approach to learn such schemes is to collect a supervised dataset of keywords-sentence pairs as a training set, but (i) it would be expensive to collect such data from users, and (ii) a static dataset would not capture a real user's natural predilection to adapt to the system BIBREF2  Another approach is to avoid supervision and jointly learn a user-system communication scheme to directly optimize the combination of efficiency and accuracy  However, learning in this way can lead to communication schemes that are uninterpretable to humans BIBREF3, BIBREF4 (see Appendix for additional related work) 
In this work, we propose a simple, unsupervised approach to an autocomplete system that is efficient, accurate, and interpretable  For interpretability, we restrict keywords to be subsequences of their source sentences based on the intuition that humans can infer most of the original meaning from a few keywords  We then apply multi-objective optimization approaches to directly control and achieve desirable tradeoffs between efficiency and accuracy 
We observe that naively optimizing a linear combination of efficiency and accuracy terms is unstable and leads to suboptimal schemes  Thus, we propose a new objective which optimizes for communication efficiency under an accuracy constraint  We show this new objective is more stable and efficient than the linear objective at all accuracy levels 
As a proof-of-concept, we build an autocomplete system within this framework which allows a user to write sentences by specifying keywords  We empirically show that our framework produces communication schemes that are 52 16% more accurate than rule-based baselines when specifying 77 37% of sentences, and 11 73% more accurate than a naive, weighted optimization approach when specifying 53 38% of sentences  Finally, we demonstrate that humans can easily adapt to the keyword-based autocomplete system and save nearly 50% of time compared to typing a full sentence in our user study 
Approach
Consider a communication game in which the goal is for a user to communicate a target sequence $x= (x_1,    , x_m)$ to a system by passing a sequence of keywords $z= (z_1,    , z_n)$  The user generates keywords $z$ using an encoding strategy $q_{\alpha }(z\mid x)$, and the system attempts to guess the target sequence $x$ via a decoding strategy $p_{\beta }(x\mid z)$ 
A good communication scheme $(q_{\alpha }, p_{\beta })$ should be both efficient and accurate  Specifically, we prefer schemes that use fewer keywords (cost), and the target sentence $x$ to be reconstructed with high probability (loss) where
Based on our assumption that humans have an intuitive sense of retaining important keywords, we restrict the set of schemes to be a (potentially noncontiguous) subsequence of the target sentence  Our hypothesis is that such subsequence schemes naturally ensure interpretability, as efficient human and machine communication schemes are both likely to involve keeping important content words 
Approach ::: Modeling with autoencoders 
To learn communication schemes without supervision, we model the cooperative communication between a user and system through an encoder-decoder framework  Concretely, we model the user's encoding strategy $q_{\alpha }(z\mid x)$ with an encoder which encodes the target sentence $x$ into the keywords $z$ by keeping a subset of the tokens  This stochastic encoder $q_{\alpha }(z\mid x)$ is defined by a model which returns the probability of each token retained in the final subsequence $z$  Then, we sample from Bernoulli distributions according to these probabilities to either keep or drop the tokens independently (see Appendix for an example)

ground_truth: Bayesian model of garg2012unsupervised as our base monolingual model

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How many sentences does the dataset contain?
Context 2:  The sentences are separated by empty newline  A sample sentence from the dataset is presented in table FIGREF13 
Dataset Statistics ::: ILPRL dataset
After much time, we received the dataset from Bal Krishna Bal, ILPRL, KU  This dataset follows standard CoNLL-2003 IOB formatBIBREF25 with POS tags  This dataset is prepared by ILPRL Lab, KU and KEIV Technologies  Few corrections like correcting the NER tags had to be made on the dataset  The statistics of both the dataset is presented in table TABREF23 
Table TABREF24 presents the total entities (PER, LOC, ORG and MISC) from both of the dataset used in our experiments  The dataset is divided into three parts with 64%, 16% and 20% of the total dataset into training set, development set and test set respectively 
Experiments
In this section, we present the details about training our neural network  The neural network architecture are implemented using PyTorch framework BIBREF26  The training is performed on a single Nvidia Tesla P100 SXM2  We first run our experiment on BiLSTM, BiLSTM-CNN, BiLSTM-CRF BiLSTM-CNN-CRF using the hyper-parameters mentioned in Table TABREF30  The training and evaluation was done on sentence-level  The RNN variants are initialized randomly from $(-\sqrt{k},\sqrt{k})$ where $k=\frac{1}{hidden\_size}$ 
First we loaded our dataset and built vocabulary using torchtext library  This eased our process of data loading using its SequenceTaggingDataset class  We trained our model with shuffled training set using Adam optimizer with hyper-parameters mentioned in table TABREF30  All our models were trained on single layer of LSTM network  We found out Adam was giving better performance and faster convergence compared to Stochastic Gradient Descent (SGD)  We chose those hyper-parameters after many ablation studies  The dropout of 0 5 is applied after LSTM layer 
For CNN, we used 30 different filters of sizes 3, 4 and 5  The embeddings of each character or grapheme involved in a given word, were passed through the pipeline of Convolution, Rectified Linear Unit and Max-Pooling  The resulting vectors were concatenated and applied dropout of 0 5 before passing into linear layer to obtain the embedding size of 30 for the given word  This resulting embedding is concatenated with word embeddings, which is again concatenated with one-hot POS vector 
Experiments ::: Tagging Scheme
Currently, for our experiments we trained our model on IO (Inside, Outside) format for both the dataset, hence the dataset does not contain any B-type annotation unlike in BIO (Beginning, Inside, Outside) scheme 
Experiments ::: Early Stopping
We used simple early stopping technique where if the validation loss does not decrease after 10 epochs, the training was stopped, else the training will run upto 100 epochs  In our experience, training usually stops around 30-50 epochs 
Experiments ::: Hyper-parameters Tuning
We ran our experiment looking for the best hyper-parameters by changing learning rate from (0,1, 0 01, 0 001, 0 0001), weight decay from [$10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$, $10^{-7}$], batch size from [1, 2, 4, 8, 16, 32, 64, 128], hidden size from [8, 16, 32, 64, 128, 256, 512 1024]  Table TABREF30 shows all other hyper-parameter used in our experiment for both of the dataset 
Experiments ::: Effect of Dropout
Figure FIGREF31 shows how we end up choosing 0 5 as dropout rate  When the dropout layer was not used, the F1 score are at the lowest  As, we slowly increase the dropout rate, the F1 score also gradually increases, however after dropout rate = 0 5, the F1 score starts falling down  Therefore, we have chosen 0 5 as dropout rate for all other experiments performed 
Evaluation
In this section, we present the details regarding evaluation and comparison of our models with other baselines 
Table TABREF25 shows the study of various embeddings and comparison among each other in OurNepali dataset  Here, raw dataset represents such dataset where post-positions are not lemmatized  We can observe that pre-trained embeddings significantly improves the score compared to randomly initialized embedding  We can deduce that Skip Gram models perform better compared CBOW models for word2vec and fasttext  Here, fastText_Pretrained represents the embedding readily available in fastText website, while other embeddings are trained on the Nepali National Corpus as mentioned in sub-section SECREF11  From this table TABREF25, we can clearly observe that model using fastText_Skip Gram embeddings outperforms all other models 
Table TABREF35 shows the model architecture comparison between all the models experimented  The features used for Stanford CRF classifier are words, letter n-grams of upto length 6, previous word and next word  This model is trained till the current function value is less than $1\mathrm {e}{-2}$  The hyper-parameters of neural network experiments are set as shown in table TABREF30  Since, word embedding of character-level and grapheme-level is random, their scores are near 
All models are evaluated using CoNLL-2003 evaluation scriptBIBREF25 to calculate entity-wise precision, recall and f1 score 
Discussion
In this paper we present that we can exploit the power of neural network to train the model to perform downstream NLP tasks like Named Entity Recognition even in Nepali language  We showed that the word vectors learned through fasttext skip gram model performs better than other word embedding because of its capability to represent sub-word and this is particularly important to capture morphological structure of words and sentences in highly inflectional like Nepali  This concept can come handy in other Devanagari languages as well because the written scripts have similar syntactical structure 
We also found out that stemming post-positions can help a lot in improving model performance because of inflectional characteristics of Nepali language  So when we separate out its inflections or morphemes, we can minimize the variations of same word which gives its root word a stronger word vector representations compared to its inflected versions 
We can clearly imply from tables TABREF23, TABREF24, and TABREF35 that we need more data to get better results because OurNepali dataset volume is almost ten times bigger compared to ILPRL dataset in terms of entities 
Conclusion and Future work
In this paper, we proposed a novel NER for Nepali language and achieved relative improvement of upto 10% and studies different factors effecting the performance of the NER for Nepali language 
We also present a neural architecture BiLSTM+CNN(grapheme-level) which turns out to be performing on par with BiLSTM+CNN(character-level) under the same configuration  We believe this will not only help Nepali language but also other languages falling under the umbrellas of Devanagari languages  Our model BiLSTM+CNN(grapheme-level) and BiLSTM+CNN(G)+POS outperforms all other model experimented in OurNepali and ILPRL dataset respectively 
Since this is the first named entity recognition research in Nepal language using neural network, there are many rooms for improvement  We believe initializing the grapheme-level embedding with fasttext embeddings might help boosting the performance, rather than randomly initializing it  In future, we plan to apply other latest techniques like BERT, ELMo and FLAIR to study its effect on low-resource language like Nepali  We also plan to improve the model using cross-lingual or multi-lingual parameter sharing techniques by jointly training with other Devanagari languages like Hindi and Bengali 
Finally, we would like to contribute our dataset to Nepali NLP community to move forward the research going on in language understanding domain  We believe there should be special committee to create and maintain such dataset for Nepali NLP and organize various competitions which would elevate the NLP research in Nepal 
Some of the future works are listed below:
Proper initialization of grapheme level embedding from fasttext embeddings 
Apply robust POS-tagger for Nepali dataset
Lemmatize the OurNepali dataset with robust and efficient lemmatizer
Improve Nepali language score with cross-lingual learning techniques
Create more dataset using Wikipedia/Wikidata framework
Acknowledgments
The authors of this paper would like to express sincere thanks to Bal Krishna Bal, Kathmandu University Professor for providing us the POS-tagged Nepali NER data 
Context 3: 
We model the autocomplete system's decoding strategy $p_{\beta }(x\mid z)$ as a probabilistic model which conditions on the keywords $z$ and returns a distribution over predictions $x$  We use a standard sequence-to-sequence model with attention and copying for the decoder, but any model architecture can be used (see Appendix for details) 
Approach ::: Multi-objective optimization 
Our goal now is to learn encoder-decoder pairs which optimally balance the communication cost and reconstruction loss  The simplest approach to balancing efficiency and accuracy is to weight $\mathrm {cost}(x, \alpha )$ and $\mathrm {loss}(x, \alpha , \beta )$ linearly using a weight $\lambda $ as follows,
where the expectation is taken over the population distribution of source sentences $x$, which is omitted to simplify notation  However, we observe that naively weighting and searching over $\lambda $ is suboptimal and highly unstable—even slight changes to the weighting results in degenerate schemes which keep all or none of its tokens  This instability motivates us to develop a new stable objective 
Our main technical contribution is to draw inspiration from the multi-objective optimization literature and view the tradeoff as a sequence of constrained optimization problems, where we minimize the expected cost subject to varying expected reconstruction error constraints $\epsilon $,
This greatly improves the stability of the training procedure  We empirically observe that the model initially keeps most of the tokens to meet the constraints, and slowly learns to drop uninformative words from the keywords to minimize the cost  Furthermore, $\epsilon $ in Eq (DISPLAY_FORM6) allows us to directly control the maximum reconstruction error of resulting schemes, whereas $\lambda $ in Eq (DISPLAY_FORM5) is not directly related to any of our desiderata 
To optimize the constrained objective, we consider the Lagrangian of Eq (DISPLAY_FORM6),
Much like the objective in Eq (DISPLAY_FORM5) we can compute unbiased gradients by replacing the expectations with their averages over random minibatches  Although gradient descent guarantees convergence on Eq (DISPLAY_FORM7) only when the objective is convex, we find that not only is the optimization stable, the resulting solution achieves better performance than the weighting approach in Eq (DISPLAY_FORM5) 
Approach ::: Optimization 
Optimization with respect to $q_{\alpha }(z\mid x)$ is challenging as $z$ is discrete, and thus, we cannot differentiate $\alpha $ through $z$ via the chain rule  Because of this, we use the stochastic REINFORCE estimate BIBREF5 as follows:
We perform joint updates on $(\alpha , \beta , \lambda )$, where $\beta $ and $\lambda $ are updated via standard gradient computations, while $\alpha $ uses an unbiased, stochastic gradient estimate where we approximate the expectation in Eq (DISPLAY_FORM9)  We use a single sample from $q_{\alpha }(z\mid x)$ and moving-average of rewards as a baseline to reduce variance 
Experiments
We evaluate our approach by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews BIBREF6 (see Appendix for details)  We quantify the efficiency of a communication scheme $(q_{\alpha },p_{\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords  The accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence 
Experiments ::: Effectiveness of constrained objective 
We first show that the linear objective in Eq (DISPLAY_FORM5) is suboptimal compared to the constrained objective in Eq (DISPLAY_FORM6)  Figure FIGREF10 compares the achievable accuracy and efficiency tradeoffs for the two objectives, which shows that the constrained objective results in more efficient schemes than the linear objective at every accuracy level (e g  11 73% more accurate at a 53 38% retention rate) 
We also observe that the linear objective is highly unstable as a function of the tradeoff parameter $\lambda $ and requires careful tuning  Even slight changes to $\lambda $ results in degenerate schemes that keep all or none of the tokens (e g  $\lambda \le 4 2$ and $\lambda \ge 4 4$)  On the other hand, the constrained objective is substantially more stable as a function of $\epsilon $ (e g  points for $\epsilon $ are more evenly spaced than $\lambda $) 
Experiments ::: Efficiency-accuracy tradeoff 
We quantify the efficiency-accuracy tradeoff compared to two rule-based baselines: Unif and Stopword  The Unif encoder randomly keeps tokens to generate keywords with the probability $\delta $

ground_truth: 3606

6946

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How are models evaluated in this human-machine communication game?
Context 3:  The Stopword encoder keeps all tokens but drops stop words (e g  `the', `a', `or') all the time ($\delta =0$) or half of the time ($\delta =0 5$)  The corresponding decoders for these encoders are optimized using gradient descent to minimize the reconstruction error (i e  $\mathrm {loss}(x, \alpha , \beta )$) 
Figure FIGREF10 shows that two baselines achieve similar tradeoff curves, while the constrained model achieves a substantial 52 16% improvement in accuracy at a 77 37% retention rate compared to Unif, thereby showing the benefits of jointly training the encoder and decoder 
Experiments ::: Robustness and analysis 
We provide additional experimental results on the robustness of learned communication schemes as well as in-depth analysis on the correlation between the retention rates of tokens and their properties, which we defer to Appendix and for space 
Experiments ::: User study 
We recruited 100 crowdworkers on Amazon Mechanical Turk (AMT) and measured completion times and accuracies for typing randomly sampled sentences from the Yelp corpus  Each user was shown alternating autocomplete and writing tasks across 50 sentences (see Appendix for user interface)  For the autocomplete task, we gave users a target sentence and asked them to type a set of keywords into the system  The users were shown the top three suggestions from the autocomplete system, and were asked to mark whether each of these three suggestions was semantically equivalent to the target sentence  For the writing task, we gave users a target sentence and asked them to either type the sentence verbatim or a sentence that preserves the meaning of the target sentence 
Table TABREF13 shows two examples of the autocomplete task and actual user-provided keywords  Each column contains a set of keywords and its corresponding top three suggestions generated by the autocomplete system with beam search  We observe that the system is likely to propose generic sentences for under-specified keywords (left column) and almost the same sentences for over-specified keywords (right column)  For properly specified keywords (middle column), the system completes sentences accordingly by adding a verb, adverb, adjective, preposition, capitalization, and punctuation 
Overall, the autocomplete system achieved high accuracy in reconstructing the keywords  Users marked the top suggestion from the autocomplete system to be semantically equivalent to the target $80 6$% of the time, and one of the top 3 was semantically equivalent $90 11$% of the time  The model also achieved a high exact match accuracy of 18 39%  Furthermore, the system was efficient, as users spent $3 86$ seconds typing keywords compared to $5 76$ seconds for full sentences on average  The variance of the typing time was $0 08$ second for keywords and $0 12$ second for full sentences, indicating that choosing and typing keywords for the system did not incur much overhead 
Experiments ::: Acknowledgments
We thank the reviewers and Yunseok Jang for their insightful comments  This work was supported by NSF CAREER Award IIS-1552635 and an Intuit Research Award 
Experiments ::: Reproducibility
All code, data and experiments are available on CodaLab at https://bit ly/353fbyn 

ground_truth: by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews

efficiency of a communication scheme $(q_{\alpha },p_{\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords, accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence