prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What metrics are used for evaluation?
Context 1: Introduction
Pre-trained models BIBREF0, BIBREF1 have received much of attention recently thanks to their impressive results in many down stream NLP tasks  Additionally, multilingual pre-trained models enable many NLP applications for other languages via zero-short cross-lingual transfer  Zero-shot cross-lingual transfer has shown promising results for rapidly building applications for low resource languages  BIBREF2 show the potential of multilingual-BERT BIBREF0 in zero-shot transfer for a large number of languages from different language families on five NLP tasks, namely, natural language inference, document classification, named entity recognition, part-of-speech tagging, and dependency parsing 
Although multilingual models are an important ingredient for enhancing language technology in many languages, recent research on improving pre-trained models puts much emphasis on English BIBREF3, BIBREF4, BIBREF5  The current state of affairs makes it difficult to translate advancements in pre-training from English to non-English languages  To our best knowledge, there are only three available multilingual pre-trained models to date: (1) the multilingual-BERT (mBERT) that supports 104 languages, (2) cross-lingual language model BIBREF6 that supports 100 languages, and (3) Language Agnostic SEntence Representations BIBREF7 that supports 93 languages  Among the three models, LASER is based on neural machine translation approach and strictly requires parallel data to train 
Do multilingual models always need to be trained from scratch? Can we transfer linguistic knowledge learned by English pre-trained models to other languages? In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way BIBREF8  As the first step, we focus on building a bilingual language model (LM) of English and a target language  Starting from a pre-trained English LM, we learn the target language specific parameters (i e , word embeddings), while keeping the encoder layers of the pre-trained English LM fixed  We then fine-tune both English and target model to obtain the bilingual LM  We apply our approach to autoencoding language models with masked language model objective and show the advantage of the proposed approach in zero-shot transfer  Our main contributions in this work are:
We propose a fast adaptation method for obtaining a bilingual BERT$_{\textsc {base}}$ of English and a target language within a day using one Tesla V100 16GB GPU 
We evaluate our bilingual LMs for six languages on two zero-shot cross-lingual transfer tasks, namely natural language inference BIBREF9 and universal dependency parsing  We show that our models offer competitive performance or even better that mBERT 
We illustrate that our bilingual LMs can serve as an excellent feature extractor in supervised dependency parsing task 
Bilingual Pre-trained LMs
We first provide some background of pre-trained language models  Let $_e$ be English word-embeddings and $\Psi ()$ be the Transformer BIBREF10 encoder with parameters $$  Let $_{w_i}$ denote the embedding of word $w_i$ (i e , $_{w_i} = _e[w_1]$)  We omit positional embeddings and bias for clarity  A pre-trained LM typically performs the following computations: (i) transform a sequence of input tokens to contextualized representations $[_{w_1},\dots ,_{w_n}] = \Psi (_{w_1}, \dots , _{w_n}; )$, and (ii) predict an output word $y_i$ at $i^{\text{th}}$ position $p(y_i | _{w_i}) \propto \exp (_{w_i}^\top _{y_i})$ 
Autoencoding LM BIBREF0 corrupts some input tokens $w_i$ by replacing them with a special token [MASK]  It then predicts the original tokens $y_i = w_i$ from the corrupted tokens  Autoregressive LM BIBREF3 predicts the next token ($y_i = w_{i+1}$) given all the previous tokens  The recently proposed XLNet model BIBREF5 is an autoregressive LM that factorizes output with all possible permutations, which shows empirical performance improvement over GPT-2 due to the ability to capture bidirectional context  Here we assume that the encoder performs necessary masking with respect to each training objective 
Given an English pre-trained LM, we wish to learn a bilingual LM for English and a given target language $f$ under a limited computational resource budget  To quickly build a bilingual LM, we directly adapt the English pre-traind model to the target model  Our approach consists of three steps  First, we initialize target language word-embeddings $_f$ in the English embedding space such that embeddings of a target word and its English equivalents are close together (§SECREF8)  Next, we create a target LM from the target embeddings and the English encoder $\Psi ()$  We then fine-tune target embeddings while keeping $\Psi ()$ fixed (§SECREF14)  Finally, we construct a bilingual LM of $_e$, $_f$, and $\Psi ()$ and fine-tune all the parameters (§SECREF15)  Figure FIGREF7 illustrates the last two steps in our approach 
Bilingual Pre-trained LMs ::: Initializing Target Embeddings
Our approach to learn the initial foreign word embeddings $_f \in ^{|V_f| \times d}$ is based on the idea of mapping the trained English word embeddings $_e \in ^{|V_e| \times d}$ to $_f$ such that if a foreign word and an English word are similar in meaning then their embeddings are similar  Borrowing the idea of universal lexical sharing from BIBREF11, we represent each foreign word embedding $_f[i] \in ^d$ as a linear combination of English word embeddings $_e[j] \in ^d$
where $_i\in ^{|V_e|}$ is a sparse vector and $\sum _j^{|V_e|} \alpha _{ij} = 1$ 
In this step of initializing foreign embeddings, having a good estimation of $$ could speed of the convergence when tuning the foreign model and enable zero-shot transfer (§SECREF5)  In the following, we discuss how to estimate $_i\;\forall i\in \lbrace 1,2, \dots , |V_f|\rbrace $ under two scenarios: (i) we have parallel data of English-foreign, and (ii) we only rely on English and foreign monolingual data 
Bilingual Pre-trained LMs ::: Initializing Target Embeddings ::: Learning from Parallel Corpus
Given an English-foreign parallel corpus, we can estimate word translation probability $p(e\,|\,f)$ for any (English-foreign) pair $(e, f)$ using popular word-alignment BIBREF12 toolkits such as fast-align BIBREF13  We then assign:
Since $_i$ is estimated from word alignment, it is a sparse vector 
Bilingual Pre-trained LMs ::: Initializing Target Embeddings ::: Learning from Monolingual Corpus
For low resource languages, parallel data may not be available  In this case, we rely only on monolingual data (e g , Wikipedias)  We estimate word translation probabilities from word embeddings of the two languages  Word vectors of these languages can be learned using fastText BIBREF14 and then are aligned into a shared space with English BIBREF15, BIBREF16  Unlike learning contextualized representations, learning word vectors is fast and computationally cheap  Given the aligned vectors $\bar{}_f$ of foreign and $\bar{}_e$ of English, we calculate the word translation matrix $\in ^{|V_f|\times |V_e|}$ as
Here, we use $\mathrm {sparsemax}$ BIBREF17 instead of softmax  Sparsemax is a sparse version of softmax and it puts zero probabilities on most of the word in the English vocabulary except few English words that are similar to a given foreign word  This property is desirable in our approach since it leads to a better initialization of the foreign embeddings 
Bilingual Pre-trained LMs ::: Fine-tuning Target Embeddings
After initializing foreign word-embeddings, we replace English word-embeddings in the English pre-trained LM with foreign word-embeddings to obtain the foreign LM  We then fine-tune only foreign word-embeddings on monolingual data  The training objective is the same as the training objective of the English pre-trained LM (i e , masked LM for BERT)  Since the trained encoder $\Psi ()$ is good at capturing association, the purpose of this step is to further optimize target embeddings such that the target LM can utilized the trained encoder for association task  For example, if the words Albert Camus presented in a French input sequence, the self-attention in the encoder more likely attends to words absurde and existentialisme once their embeddings are tuned 
Bilingual Pre-trained LMs ::: Fine-tuning Bilingual LM
We create a bilingual LM by plugging foreign language specific parameters to the pre-trained English LM (Figure FIGREF7)  The new model has two separate embedding layers and output layers, one for English and one for foreign language  The encoder layer in between is shared  We then fine-tune this model using English and foreign monolingual data  Here, we keep tuning the model on English to ensure that it does not forget what it has learned in English and that we can use the resulting model for zero-shot transfer (§SECREF3)  In this step, the encoder parameters are also updated so that in can learn syntactic aspects (i e , word order, morphological agreement) of the target languages 
Zero-shot Experiments
We build our bilingual LMs, named RAMEN, starting from BERT$_{\textsc {base}}$, BERT$_{\textsc {large}}$, RoBERTa$_{\textsc {base}}$, and RoBERTa$_{\textsc {large}}$ pre-trained models  Using BERT$_{\textsc {base}}$ allows us to compare the results with mBERT model  Using BERT$_{\textsc {large}}$ and RoBERTa allows us to investigate whether the performance of the target LM correlates with the performance of the source LM  We evaluate our models on two cross-lingual zero-shot tasks: (1) Cross-lingual Natural Language Inference (XNLI) and (2) dependency parsing 
Zero-shot Experiments ::: Data
We evaluate our approach for six target languages: French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi)  These languages belong to four different language families  French, Russian, and Hindi are Indo-European languages, similar to English  Arabic, Chinese, and Vietnamese belong to Afro-Asiatic, Sino-Tibetan, and Austro-Asiatic family respectively  The choice of the six languages also reflects different training conditions depending on the amount of monolingual data  French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource 
For experiments that use parallel data to initialize foreign specific parameters, we use the same datasets in the work of BIBREF6  Specifically, we use United Nations Parallel Corpus BIBREF18 for en-ru, en-ar, en-zh, and en-fr  We collect en-hi parallel data from IIT Bombay corpus BIBREF19 and en-vi data from OpenSubtitles 2018  For experiments that use only monolingual data to initialize foreign parameters, instead of training word-vectors from the scratch, we use the pre-trained word vectors from fastText BIBREF14 to estimate word translation probabilities (Eq  DISPLAY_FORM13)  We align these vectors into a common space using orthogonal Procrustes BIBREF20, BIBREF15, BIBREF16  We only use identical words between the two languages as the supervised signal  We use WikiExtractor to extract extract raw sentences from Wikipedias as monolingual data for fine-tuning target embeddings and bilingual LMs (§SECREF15)  We do not lowercase or remove accents in our data preprocessing pipeline 
We tokenize English using the provided tokenizer from pre-trained models  For target languages, we use fastBPE to learn 30,000 BPE codes and 50,000 codes when transferring from BERT and RoBERTa respectively  We truncate the BPE vocabulary of foreign languages to match the size of the English vocabulary in the source models  Precisely, the size of foreign vocabulary is set to 32,000 when transferring from BERT and 50,000 when transferring from RoBERTa 
We use XNLI dataset BIBREF9 for classification task and Universal Dependencies v2 4 BIBREF21 for parsing task  Since a language might have more than one treebank in Universal Dependencies, we use the following treebanks: en_ewt (English), fr_gsd (French), ru_syntagrus (Russian) ar_padt (Arabic), vi_vtb (Vietnamese), hi_hdtb (Hindi), and zh_gsd (Chinese) 
Zero-shot Experiments ::: Data ::: Remark on BPE
BIBREF22 show that sharing subwords between languages improves alignments between embedding spaces  BIBREF2 observe a strong correlation between the percentage of overlapping subwords and mBERT's performances for cross-lingual zero-shot transfer  However, in our current approach, subwords between source and target are not shared  A subword that is in both English and foreign vocabulary has two different embeddings 
Zero-shot Experiments ::: Estimating translation probabilities
Since pre-trained models operate on subword level, we need to estimate subword translation probabilities  Therefore, we subsample 2M sentence pairs from each parallel corpus and tokenize the data into subwords before running fast-align BIBREF13 
Estimating subword translation probabilities from aligned word vectors requires an additional processing step since the provided vectors from fastText are not at subword level  We use the following approximation to obtain subword vectors: the vector $_s$ of subword $s$ is the weighted average of all the aligned word vectors $_{w_i}$ that have $s$ as an subword
where $p(w_j)$ is the unigram probability of word $w_j$ and $n_s = \sum _{w_j:\, s\in w_j} p(w_j)$  We take the top 50,000 words in each aligned word-vectors to compute subword vectors 
In both cases, not all the words in the foreign vocabulary can be initialized from the English word-embeddings  Those words are initialized randomly from a Gaussian $\mathcal {N}(0, {1}{d^2})$ 
Zero-shot Experiments ::: Hyper-parameters
In all the experiments, we tune RAMEN$_{\textsc {base}}$ for 175,000 updates and RAMEN$_{\textsc {large}}$ for 275,000 updates where the first 25,000 updates are for language specific parameters  The sequence length is set to 256  The mini-batch size are 64 and 24 when tuning language specific parameters using RAMEN$_{\textsc {base}}$ and RAMEN$_{\textsc {large}}$ respectively  For tuning bilingual LMs, we use a mini-batch size of 64 for RAMEN$_{\textsc {base}}$ and 24 for RAMEN$_{\textsc {large}}$ where half of the batch are English sequences and the other half are foreign sequences  This strategy of balancing mini-batch has been used in multilingual neural machine translation BIBREF23, BIBREF24 
We optimize RAMEN$_{\textsc {base}}$ using Lookahead optimizer BIBREF25 wrapped around Adam with the learning rate of $10^{-4}$, the number of fast weight updates $k=5$, and interpolation parameter $\alpha =0 5$  We choose Lookahead optimizer because it has been shown to be robust to the initial parameters of the based optimizer (Adam)  For Adam optimizer, we linearly increase the learning rate from $10^{-7}$ to $10^{-4}$ in the first 4000 updates and then follow an inverse square root decay  All RAMEN$_{\textsc {large}}$ models are optimized with Adam due to memory limit 
When fine-tuning RAMEN on XNLI and UD, we use a mini-batch size of 32, Adam's learning rate of $10^{-5}$  The number of epochs are set to 4 and 50 for XNLI and UD tasks respectively  All experiments are carried out on a single Tesla V100 16GB GPU  Each RAMEN$_{\textsc {base}}$ model is trained within a day and each RAMEN$_{\textsc {large}}$ is trained within two days 
Results
In this section, we present the results of out models for two zero-shot cross lingual transfer tasks: XNLI and dependency parsing 
Results ::: Cross-lingual Natural Language Inference
Table TABREF32 shows the XNLI test accuracy  For reference, we also include the scores from the previous work, notably the state-of-the-art system XLM BIBREF6  Before discussing the results, we spell out that the fairest comparison in this experiment is the comparison between mBERT and RAMEN$_{\textsc {base}}$+BERT trained with monolingual only 
We first discuss the transfer results from BERT  Initialized from fastText vectors, RAMEN$_{\textsc {base}}$ slightly outperforms mBERT by 1 9 points on average and widen the gap of 3 3 points on Arabic  RAMEN$_{\textsc {base}}$ gains extra 0 8 points on average when initialized from parallel data  With triple number of parameters, RAMEN$_{\textsc {large}}$ offers an additional boost in term of accuracy and initialization with parallel data consistently improves the performance  It has been shown that BERT$_{\textsc {large}}$ significantly outperforms BERT$_{\textsc {base}}$ on 11 English NLP tasks BIBREF0, the strength of BERT$_{\textsc {large}}$ also shows up when adapted to foreign languages 
Transferring from RoBERTa leads to better zero-shot accuracies  With the same initializing condition, RAMEN$_{\textsc {base}}$+RoBERTa outperforms RAMEN$_{\textsc {base}}$+BERT on average by 2 9 and 2 3 points when initializing from monolingual and parallel data respectively  This result show that with similar number of parameters, our approach benefits from a better English pre-trained model  When transferring from RoBERTa$_{\textsc {large}}$, we obtain state-of-the-art results for five languages 
Currently, RAMEN only uses parallel data to initialize foreign embeddings  RAMEN can also exploit parallel data through translation objective proposed in XLM  We believe that by utilizing parallel data during the fine-tuning of RAMEN would bring additional benefits for zero-shot tasks  We leave this exploration to future work  In summary, starting from BERT$_{\textsc {base}}$, our approach obtains competitive bilingual LMs with mBERT for zero-shot XNLI  Our approach shows the accuracy gains when adapting from a better pre-trained model 
Results ::: Universal Dependency Parsing
We build on top of RAMEN a graph-based dependency parser BIBREF27  For the purpose of evaluating the contextual representations learned by our model, we do not use part-of-speech tags  Contextualized representations are directly fed into Deep-Biaffine layers to predict arc and label scores  Table TABREF34 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing 
We first look at the fairest comparison between mBERT and monolingually initialized RAMEN$_{\textsc {base}}$+BERT  The latter outperforms the former on five languages except Arabic  We observe the largest gain of +5 2 LAS for French  Chinese enjoys +3 1 LAS from our approach  With similar architecture (12 or 24 layers) and initialization (using monolingual or parallel data), RAMEN+RoBERTa performs better than RAMEN+BERT for most of the languages  Arabic and Hindi benefit the most from bigger models  For the other four languages, RAMEN$_{\textsc {large}}$ renders a modest improvement over RAMEN$_{\textsc {base}}$ 
Analysis ::: Impact of initialization
Initializing foreign embeddings is the backbone of our approach  A good initialization leads to better zero-shot transfer results and enables fast adaptation  To verify the importance of a good initialization, we train a RAMEN$_{\textsc {base}}$+RoBERTa with foreign word-embeddings are initialized randomly from $\mathcal {N}(0, {1}{d^2})$  For a fair comparison, we use the same hyper-parameters in §SECREF27  Table TABREF36 shows the results of XNLI and UD parsing of random initialization  In comparison to the initialization using aligned fastText vectors, random initialization decreases the zero-shot performance of RAMEN$_{\textsc {base}}$ by 15 9% for XNLI and 27 8 points for UD parsing on average  We also see that zero-shot parsing of SOV languages (Arabic and Hindi) suffers random initialization 
Analysis ::: Are contextual representations from RAMEN also good for supervised parsing?
All the RAMEN models are built from English and tuned on English for zero-shot cross-lingual tasks  It is reasonable to expect RAMENs do well in those tasks as we have shown in our experiments  But are they also a good feature extractor for supervised tasks? We offer a partial answer to this question by evaluating our model for supervised dependency parsing on UD datasets 
We used train/dev/test splits provided in UD to train and evaluate our RAMEN-based parser  Table TABREF38 summarizes the results (LAS) of our supervised parser  For a fair comparison, we choose mBERT as the baseline and all the RAMEN models are initialized from aligned fastText vectors  With the same architecture of 12 Transformer layers, RAMEN$_{\textsc {base}}$+BERT performs competitive to mBERT and outshines mBERT by +1 2 points for Vietnamese  The best LAS results are obtained by RAMEN$_{\textsc {large}}$+RoBERTa with 24 Transformer layers  Overall, our results indicate the potential of using contextual representations from RAMEN for supervised tasks 
Analysis ::: How does linguistic knowledge transfer happen through each training stages?
We evaluate the performance of RAMEN+RoBERTa$_{\textsc {base}}$ (initialized from monolingual data) at each training steps: initialization of word embeddings (0K update), fine-tuning target embeddings (25K), and fine-tuning the model on both English and target language (at each 25K updates)  The results are presented in Figure FIGREF40 
Without fine-tuning, the average accuracy of XLNI is 39 7% for a three-ways classification task, and the average LAS score is 3 6 for dependency parsing  We see the biggest leap in the performance after 50K updates  While semantic similarity task profits significantly at 25K updates of the target embeddings, syntactic task benefits with further fine-tuning the encoder  This is expected since the target languages might exhibit different syntactic structures than English and fine-tuning encoder helps to capture language specific structures  We observe a substantial gain of 19-30 LAS for all languages except French after 50K updates 
Language similarities have more impact on transferring syntax than semantics  Without tuning the English encoder, French enjoys 50 3 LAS for being closely related to English, whereas Arabic and Hindi, SOV languages, modestly reach 4 2 and 6 4 points using the SVO encoder  Although Chinese has SVO order, it is often seen as head-final while English is strong head-initial  Perhaps, this explains the poor performance for Chinese 
Limitations
While we have successfully adapted autoencoding pre-trained LMs from English to other languages, the question whether our approach can also be applied for autoregressive LM such as XLNet still remains  We leave the investigation to future work 
Conclusions
In this work, we have presented a simple and effective approach for rapidly building a bilingual LM under a limited computational budget  Using BERT as the starting point, we demonstrate that our approach produces better than mBERT on two cross-lingual zero-shot sentence classification and dependency parsing  We find that the performance of our bilingual LM, RAMEN, correlates with the performance of the original pre-trained English models  We also find that RAMEN is also a powerful feature extractor in supervised dependency parsing  Finally, we hope that our work sparks of interest in developing fast and effective methods for transferring pre-trained English models to other languages 
Context 2: Introduction
Named Entity Recognition (NER) is a foremost NLP task to label each atomic elements of a sentence into specific categories like "PERSON", "LOCATION", "ORGANIZATION" and othersBIBREF0  There has been an extensive NER research on English, German, Dutch and Spanish language BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, and notable research on low resource South Asian languages like HindiBIBREF6, IndonesianBIBREF7 and other Indian languages (Kannada, Malayalam, Tamil and Telugu)BIBREF8  However, there has been no study on developing neural NER for Nepali language  In this paper, we propose a neural based Nepali NER using latest state-of-the-art architecture based on grapheme-level which doesn't require any hand-crafted features and no data pre-processing 
Recent neural architecture like BIBREF1 is used to relax the need to hand-craft the features and need to use part-of-speech tag to determine the category of the entity  However, this architecture have been studied for languages like English, and German and not been applied to languages like Nepali which is a low resource language i e limited data set to train the model  Traditional methods like Hidden Markov Model (HMM) with rule based approachesBIBREF9,BIBREF10, and Support Vector Machine (SVM) with manual feature-engineeringBIBREF11 have been applied but they perform poor compared to neural  However, there has been no research in Nepali NER using neural network  Therefore, we created the named entity annotated dataset partly with the help of Dataturk to train a neural model  The texts used for this dataset are collected from various daily news sources from Nepal around the year 2015-2016 
Following are our contributions:
We present a novel Named Entity Recognizer (NER) for Nepali language  To best of our knowledge we are the first to propose neural based Nepali NER 
As there are not good quality dataset to train NER we release a dataset to support future research
We perform empirical evaluation of our model with state-of-the-art models with relative improvement of upto 10%
In this paper, we present works similar to ours in Section SECREF2  We describe our approach and dataset statistics in Section SECREF3 and SECREF4, followed by our experiments, evaluation and discussion in Section SECREF5, SECREF6, and SECREF7  We conclude with our observations in Section SECREF8 
To facilitate further research our code and dataset will be made available at github com/link-yet-to-be-updated
Related Work
There has been a handful of research on Nepali NER task based on approaches like Support Vector Machine and gazetteer listBIBREF11 and Hidden Markov Model and gazetteer listBIBREF9,BIBREF10 
BIBREF11 uses SVM along with features like first word, word length, digit features and gazetteer (person, organization, location, middle name, verb, designation and others)  It uses one vs rest classification model to classify each word into different entity classes  However, it does not the take context word into account while training the model  Similarly, BIBREF9 and BIBREF10 uses Hidden Markov Model with n-gram technique for extracting POS-tags  POS-tags with common noun, proper noun or combination of both are combined together, then uses gazetteer list as look-up table to identify the named entities 
Researchers have shown that the neural networks like CNNBIBREF12, RNNBIBREF13, LSTMBIBREF14, GRUBIBREF15 can capture the semantic knowledge of language better with the help of pre-trained embbeddings like word2vecBIBREF16, gloveBIBREF17 or fasttextBIBREF18 
Similar approaches has been applied to many South Asian languages like HindiBIBREF6, IndonesianBIBREF7, BengaliBIBREF19 and In this paper, we present the neural network architecture for NER task in Nepali language, which doesn't require any manual feature engineering nor any data pre-processing during training  First we are comparing BiLSTMBIBREF14, BiLSTM+CNNBIBREF20, BiLSTM+CRFBIBREF1, BiLSTM+CNN+CRFBIBREF2 models with CNN modelBIBREF0 and Stanford CRF modelBIBREF21  Secondly, we show the comparison between models trained on general word embeddings, word embedding + character-level embedding, word embedding + part-of-speech(POS) one-hot encoding and word embedding + grapheme clustered or sub-word embeddingBIBREF22  The experiments were performed on the dataset that we created and on the dataset received from ILPRL lab  Our extensive study shows that augmenting word embedding with character or grapheme-level representation and POS one-hot encoding vector yields better results compared to using general word embedding alone 
Approach
In this section, we describe our approach in building our model  This model is partly inspired from multiple models BIBREF20,BIBREF1, andBIBREF2
Approach ::: Bidirectional LSTM
We used Bi-directional LSTM to capture the word representation in forward as well as reverse direction of a sentence  Generally, LSTMs take inputs from left (past) of the sentence and computes the hidden state  However, it is proven beneficialBIBREF23 to use bi-directional LSTM, where, hidden states are computed based from right (future) of sentence and both of these hidden states are concatenated to produce the final output as $h_t$=[$\overrightarrow{h_t}$;$\overleftarrow{h_t}$], where $\overrightarrow{h_t}$, $\overleftarrow{h_t}$ = hidden state computed in forward and backward direction respectively 
Approach ::: Features ::: Word embeddings
We have used Word2Vec BIBREF16, GloVe BIBREF17 and FastText BIBREF18 word vectors of 300 dimensions  These vectors were trained on the corpus obtained from Nepali National Corpus  This pre-lemmatized corpus consists of 14 million words from books, web-texts and news papers  This corpus was mixed with the texts from the dataset before training CBOW and skip-gram version of word2vec using gensim libraryBIBREF24  This trained model consists of vectors for 72782 unique words 
Light pre-processing was performed on the corpus before training it  For example, invalid characters or characters other than Devanagari were removed but punctuation and numbers were not removed  We set the window context at 10 and the rare words whose count is below 5 are dropped  These word embeddings were not frozen during the training session because fine-tuning word embedding help achieve better performance compared to frozen oneBIBREF20 
We have used fasttext embeddings in particular because of its sub-word representation ability, which is very useful in highly inflectional language as shown in Table TABREF25  We have trained the word embedding in such a way that the sub-word size remains between 1 and 4  We particularly chose this size because in Nepali language a single letter can also be a word, for example e, t, C, r, l, n, u and a single character (grapheme) or sub-word can be formed after mixture of dependent vowel signs with consonant letters for example, C + O + = CO, here three different consonant letters form a single sub-word 
The two-dimensional visualization of an example word npAl is shown in FIGREF14  Principal Component Analysis (PCA) technique was used to generate this visualization which helps use to analyze the nearest neighbor words of a given sample word  84 and 104 nearest neighbors were observed using word2vec and fasttext embedding respectively on the same corpus 
Approach ::: Features ::: Character-level embeddings
BIBREF20 and BIBREF2 successfully presented that the character-level embeddings, extracted using CNN, when combined with word embeddings enhances the NER model performance significantly, as it is able to capture morphological features of a word  Figure FIGREF7 shows the grapheme-level CNN used in our model, where inputs to CNN are graphemes  Character-level CNN is also built in similar fashion, except the inputs are characters  Grapheme or Character -level embeddings are randomly initialized from [0,1] with real values with uniform distribution of dimension 30 
Approach ::: Features ::: Grapheme-level embeddings
Grapheme is atomic meaningful unit in writing system of any languages  Since, Nepali language is highly morphologically inflectional, we compared grapheme-level representation with character-level representation to evaluate its effect  For example, in character-level embedding, each character of a word npAl results into n + + p + A + l has its own embedding  However, in grapheme level, a word npAl is clustered into graphemes, resulting into n + pA + l  Here, each grapheme has its own embedding  This grapheme-level embedding results good scores on par with character-level embedding in highly inflectional languages like Nepali, because graphemes also capture syntactic information similar to characters  We created grapheme clusters using uniseg package which is helpful in unicode text segmentations 
Approach ::: Features ::: Part-of-speech (POS) one hot encoding
We created one-hot encoded vector of POS tags and then concatenated with pre-trained word embeddings before passing it to BiLSTM network  A sample of data is shown in figure FIGREF13 
Dataset Statistics ::: OurNepali dataset
Since, we there was no publicly available standard Nepali NER dataset and did not receive any dataset from the previous researchers, we had to create our own dataset  This dataset contains the sentences collected from daily newspaper of the year 2015-2016  This dataset has three major classes Person (PER), Location (LOC) and Organization (ORG)  Pre-processing was performed on the text before creation of the dataset, for example all punctuations and numbers besides ',', '-', '|' and ' ' were removed  Currently, the dataset is in standard CoNLL-2003 IO formatBIBREF25 
Since, this dataset is not lemmatized originally, we lemmatized only the post-positions like Ek, kO, l, mA, m, my, jF, sg, aEG which are just the few examples among 299 post positions in Nepali language  We obtained these post-positions from sanjaalcorps and added few more to match our dataset  We will be releasing this list in our github repository  We found out that lemmatizing the post-positions boosted the F1 score by almost 10% 
In order to label our dataset with POS-tags, we first created POS annotated dataset of 6946 sentences and 16225 unique words extracted from POS-tagged Nepali National Corpus and trained a BiLSTM model with 95 14% accuracy which was used to create POS-tags for our dataset 
The dataset released in our github repository contains each word in newline with space separated POS-tags and Entity-tags
Context 3: Introduction
How humans process language has become increasingly relevant in natural language processing since physiological data during language understanding is more accessible and recorded with less effort  In this work, we focus on eye-tracking and electroencephalography (EEG) recordings to capture the reading process  On one hand, eye movement data provides millisecond-accurate records about where humans look when they are reading, and is highly correlated with the cognitive load associated with different stages of text processing  On the other hand, EEG records electrical brain activity across the scalp and is a direct measure of physiological processes, including language processing  The combination of both measurement methods enables us to study the language understanding process in a more natural setting, where participants read full sentences at a time, in their own speed  Eye-tracking then permits us to define exact word boundaries in the timeline of a subject reading a sentence, allowing the extraction of brain activity signals for each word 
Human cognitive language processing data is immensely useful for NLP: Not only can it be leveraged to improve NLP applications (e g  barrett2016weakly for part-of-speech tagging or klerke2016improving for sentence compression), but also to evaluate state-of-the-art machine learning systems  For example, hollenstein2019cognival evaluate word embeddings, or schwartz2019inducing fine-tune language models with brain-relevant bias 
Additionally, the availability of labelled data plays a crucial role in all supervised machine learning applications  Physiological data can be used to understand and improve the labelling process (e g  tokunaga2017eye), and, for instance, to build cost models for active learning scenarios BIBREF0  Is it possible to replace this expensive manual work with models trained on physiological activity data recorded from humans while reading? That is to say, can we find and extract relevant aspects of text understanding and annotation directly from the source, i e  eye-tracking and brain activity signals during reading?
Motivated by these questions and our previously released dataset, ZuCo 1 0 BIBREF1, we developed this new corpus, where we specifically aim to collect recordings during natural reading as well as during annotation 
We provide the first dataset of simultaneous eye movement and brain activity recordings to analyze and compare normal reading to task-specific reading during annotation  The Zurich Cognitive Language Processing Corpus (ZuCo) 2 0, including raw and preprocessed eye-tracking and electroencephalography (EEG) data of 18 subjects, as well as the recording and preprocessing scripts, is publicly available at https://osf io/2urht/  It contains physiological data of each subject reading 739 English sentences from Wikipedia (see example in Figure FIGREF1)  We want to highlight the re-use potential of this data  In addition to the psycholinguistic motivation, this corpus is especially tailored for training and evaluating machine learning algorithms for NLP purposes  We conduct a detailed technical validation of the data as proof of the quality of the recordings 
Related Work
Some eye-tracking corpora of natural reading (e g  the Dundee BIBREF2, Provo BIBREF3 and GECO corpus BIBREF4), and a few EEG corpora (for example, the UCL corpus BIBREF5) are available  It has been shown that this type of cognitive processing data is useful for improving and evaluating NLP methods (e g  barrett2018sequence,hollenstein2019cognival, hale2018finding)  However, before the Zurich Cognitive Language Processing Corpus (ZuCo 1 0), there was no available data for simultaneous eye-tracking and EEG recordings of natural reading  dimigen2011coregistration studied the linguistic effects of eye movements and EEG co-registration in natural reading and showed that they accurately represent lexical processing  Moreover, the simultaneous recordings are crucial to extract word-level brain activity signals 
While the above mentioned studies analyze and leverage natural reading, some NLP work has used eye-tracking during annotation (but, as of yet, not EEG data)  mishra2016predicting and joshi2014measuring recorded eye-tracking during binary sentiment annotation (positive/negative)  This data was used to determine the annotation complexity of the text passages based on eye movement metrics and for sarcasm detection BIBREF6  Moreover, eye-tracking has been used to analyze the word sense annotation process in Hindi BIBREF7, named entity annotation in Japanese BIBREF8, and to leverage annotator gaze behaviour for coreference resolution BIBREF9  Finally, tomanek2010cognitive used eye-tracking data during entity annotation to build a cost model for active learning  However, until now there is no available data or research that analyzes the differences in the human processing of normal reading versus annotation 
Related Work ::: ZuCo1 0
In previous work, we recorded a first dataset of simultaneous eye-tracking and EEG during natural reading BIBREF1  ZuCo 1 0 consists of three reading tasks, two of which contain very similar reading material and experiments as presented in the current work  However, the main difference and reason for recording ZuCo 2 0, consists in the experiment procedure  For ZuCo 1 0 the normal reading and task-specific reading paradigms were recorded in different sessions on different days  Therefore, the recorded data is not appropriate as a means of comparison between natural reading and annotation, since the differences in the brain activity data might result mostly from the different sessions due to the sensitivity of EEG  This, and extending the dataset with more sentences and more subjects, were the main factors for recording the current corpus  We purposefully maintained an overlap of some sentences between both datasets to allow additional analyses (details are described in Section SECREF7) 
Corpus Construction
In this section we describe the contents and experimental design of the ZuCo 2 0 corpus 
Corpus Construction ::: Participants
We recorded data from 19 participants and discarded the data of one of them due to technical difficulties with the eye-tracking calibration  Hence, we share the data of 18 participants  All participants are healthy adults (mean age = 34 (SD=8 3), 10 females)  Their native language is English, originating from Australia, Canada, UK, USA or South Africa  Two participants are left-handed and three participants wear glasses for reading  Details on subject demographics can be found in Table TABREF4  All participants gave written consent for their participation and the re-use of the data prior to the start of the experiments  The study was approved by the Ethics Commission of the University of Zurich 
Corpus Construction ::: Reading materials
During the recording session, the participants read 739 sentences that were selected from the Wikipedia corpus provided by culotta2006integrating  This corpus was chosen because it provides annotations of semantic relations  We included seven of the originally defined relation types: political_affiliation, education, founder, wife/husband, job_title, nationality, and employer  The sentences were chosen in the same length range as ZuCo 1 0, and with similar Flesch reading ease scores  The dataset statistics are shown in Table TABREF2 
Of the 739 sentences, the participants read 349 sentences in a normal reading paradigm, and 390 sentences in a task-specific reading paradigm, in which they had to determine whether a certain relation type occurred in the sentence or not  Table TABREF3 shows the distribution of the different relation types in the sentences of the task-specific annotation paradigm 
Purposefully, there are 63 duplicates between the normal reading and the task-specific sentences (8% of all sentences)  The intention of these duplicate sentences is to provide a set of sentences read twice by all participants with a different task in mind  Hence, this enables the comparison of eye-tracking and brain activity data when reading normally and when annotating specific relations (see examples in Section SECREF4) 
Furthermore, there is also an overlap in the sentences between ZuCo 1 0 and ZuCo 2 0  100 normal reading and 85 task-specific sentences recorded for this dataset were already recorded in ZuCo 1 0  This allows for comparisons between the different recording procedures (i e  session-specific effects) and between more participants (subject-specific effects) 
Corpus Construction ::: Experimental design
As mentioned above, we recorded two different reading tasks for the ZuCo 2 0 dataset  During both tasks the participants were able to read in their own speed, using a control pad to move to the next sentence and to answer the control questions, which allowed for natural reading  Since each subject reads at their own personal pace, the reading speed between varies between subjects  Table TABREF4 shows the average reading speed for each task, i e  the average number of seconds a subject spends per sentence before switching to the next one 
All 739 sentences were recorded in a single session for each participant  The duration of the recording sessions was between 100 and 180 minutes, depending on the time required to set up and calibrate the devices, and the personal reading speed of the participants 
We recorded 14 blocks of approx  50 sentences, alternating between tasks: 50 sentences of normal reading, followed by 50 sentences of task-specific reading  The order of blocks and sentences within blocks was identical for all subjects  Each sentence block was preceded by a practice round of three sentences 
Corpus Construction ::: Experimental design ::: Normal reading (NR)
In the first task, participants were instructed to read the sentences naturally, without any specific task other than comprehension  Participants were told to read the sentences normally without any special instructions  Figure FIGREF8 (left) shows an example sentence as it was depicted on the screen during recording  As shown in Figure FIGREF8 (middle), the control condition for this task consisted of single-choice questions about the content of the previous sentence  12% of randomly selected sentences were followed by such a comprehension question with three answer options on a separate screen 
Corpus Construction ::: Experimental design ::: Task-specific reading (TSR)
In the second task, the participants were instructed to search for a specific relation in each sentence they read  Instead of comprehension questions, the participants had to decide for each sentence whether it contains the relation or not, i e  they were actively annotating each sentence  Figure FIGREF8 (right) shows an example screen for this task  17% of the sentences did not include the relation type and were used as control conditions  All sentences within one block involved the same relation type  The blocks started with a practice round, which described the relation and was followed by three sample sentences, so that the participants would be familiar with the respective relation type 
Corpus Construction ::: Linguistic assessment
As a linguistic assessment, the vocabulary and language proficiency of the participants was tested with the LexTALE test (Lexical Test for Advanced Learners of English, lemhofer2012introducing)  This is an unspeeded lexical decision task designed for intermediate to highly proficient language users  The average LexTALE score over all participants was 88 54%  Moreover, we also report the scores the participants achieved with their answers to the reading comprehension control questions and their relation annotations  The detailed scores for all participants are also presented in Table TABREF4 
Corpus Construction ::: Data acquisition
Data acquisition took place in a sound-attenuated and dark experiment room  Participants were seated at a distance of 68cm from a 24-inch monitor with a resolution of 800x600 pixels  A stable head position was ensured via a chin rest  Participants were instructed to stay as still as possible during the tasks to avoid motor EEG artifacts  Participants were also offered snacks and water during the breaks and were encouraged to rest  All sentences were presented at the same position on the screen and could span multiple lines  The sentences were presented in black on a light grey background with font size 20-point Arial, resulting in a letter height of 0 8 mm  The experiment was programmed in MATLAB 2016b BIBREF10, using PsychToolbox BIBREF11  Participants completed the tasks sitting alone in the room, while two research assistants were monitoring their progress in the adjoining room  All recording scripts including detailed participant instructions are available alongside the data 
Corpus Construction ::: Data acquisition ::: Eye-tracking acquisition
Eye position and pupil size were recorded with an infrared video-based eye tracker (EyeLink 1000 Plus, SR Research) at a sampling rate of 500 Hz  The eye tracker was calibrated with a 9-point grid at the beginning of the session and re-validated before each block of sentences 
Corpus Construction ::: Data acquisition ::: EEG acquisition
High-density EEG data were recorded at a sampling rate of 500 Hz with a bandpass of 0 1 to 100 Hz, using a 128-channel EEG Geodesic Hydrocel system (Electrical Geodesics)  The recording reference was set at electrode Cz  The head circumference of each participant was measured to select an appropriately sized EEG net  To ensure good contact, the impedance of each electrode was checked prior to recording, and was kept below 40 kOhm  Electrode impedance levels were checked after every third block of 50 sentences (approx  every 30 mins) and reduced if necessary 
Corpus Construction ::: Preprocessing and feature extraction ::: Eye-tracking
The eye-tracking data consists of (x,y) gaze location entries for all individual fixations (Figure FIGREF1b)  Coordinates were given in pixels with respect to the monitor coordinates (the upper left corner of the screen was (0,0) and down/right was positive)  We provide this raw data as well as various engineered eye-tracking features  For this feature extraction only fixations within the boundaries of each displayed word were extracted  Data points distinctly not associated with reading (minimum distance of 50 pixels to the text) were excluded  Additionally, fixations shorter than 100 ms were excluded from the analyses, because these are unlikely to reflect fixations relevant for reading BIBREF12  On the basis of the GECO and ZuCo 1 0 corpora, we extracted the following features: (i) gaze duration (GD), the sum of all fixations on the current word in the first-pass reading before the eye moves out of the word; (ii) total reading time (TRT), the sum of all fixation durations on the current word, including regressions; (iii) first fixation duration (FFD), the duration of the first fixation on the prevailing word; (iv) single fixation duration (SFD), the duration of the first and only fixation on the current word; and (v) go-past time (GPT), the sum of all fixations prior to progressing to the right of the current word, including regressions to previous words that originated from the current word  For each of these eye-tracking features we additionally computed the pupil size  Furthermore, we extracted the number of fixations and mean pupil size for each word and sentence 
Corpus Construction ::: Preprocessing and feature extraction ::: EEG
The EEG data shared in this project are available as raw data, but also preprocessed with Automagic (version 1 4 6, pedroni2019automagic), a tool for automatic EEG data cleaning and validation  105 EEG channels (i e  electrodes) were used from the scalp recordings  9 EOG channels were used for artifact removal and additional 14 channels lying mainly on the neck and face were discarded before data analysis  Bad channels were identified and interpolated  We used the Multiple Artifact Rejection Algorithm (MARA), a supervised machine learning algorithm that evaluates ICA components, for automatic artifact rejection  MARA has been trained on manual component classifications, and thus captures a wide range of artifacts  MARA is especially effective at detecting and removing eye and muscle artifact components  The effect of this preprocessing can be seen in Figure FIGREF1d 
After preprocessing, we synchronized the EEG and eye-tracking data to enable EEG analyses time-locked to the onsets of fixations  To compute oscillatory power measures, we band-pass filtered the continuous EEG signals across an entire reading task for five different frequency bands resulting in a time-series for each frequency band  The independent frequency bands were determined as follows: theta$_1$ (4–6 Hz), theta$_2$ (6 5–8 Hz), alpha$_1$ (8 5–10 Hz), alpha$_2$ (10 5–13 Hz), beta$_1$ (13 5–18 Hz), beta$_2$ (18 5–30 Hz), gamma$_1$ (30 5–40 Hz), and gamma$_2$ (40–49 5 Hz)  We then applied a Hilbert transformation to each of these time-series  We specifically chose the Hilbert transformation to maintain the temporal information of the amplitude of the frequency bands, to enable the power of the different frequencies for time segments defined through the fixations from the eye-tracking recording  Thus, for each eye-tracking feature we computed the corresponding EEG feature in each frequency band  Furthermore, we extracted sentence-level EEG features by calculating the power in each frequency band, and additionally, the difference of the power spectra between frontal left and right homologue electrodes pairs  For each eye-tracking based EEG feature, all channels were subject to an artifact rejection criterion of $90\mu V$ to exclude trials with transient noise 
Data Validation
The aim of the technical validation of the data is to guarantee good recording quality and to replicate findings of previous studies investigating co-registration of EEG and eye movement data during natural reading tasks (e g  dimigen2011coregistration)  We also compare the results to ZuCo 1 0 BIBREF1, which allows a more direct comparison due to the analogous recording procedure 
Data Validation ::: Eye-tracking
We validated the recorded eye-tracking data by analyzing the fixations made by all subjects through their reading speed and omission rate on sentence level  The omission rate is defined as the percentage of words that is not fixated in a sentence  Figure FIGREF10 (middle) shows the mean reading speed over all subjects, measured in seconds per sentence and Figure FIGREF10 (right) shows the mean omission rates aggregated over all subjects for each task  Clearly, the participants made less fixations during the task-specific reading, which lead to faster reading speed 
Moreover, we corroborated these sentence-level metrics by visualizing the skipping proportion on word level (Figure FIGREF13)  The skipping proportion is the average rate of words being skipped (i e  not being fixated) in a sentence  As expected, this also increases in the task-specific reading 
Although the reading material is from the same source and of the same length range (see Figure FIGREF10 (left)), in the first task (NR) passive reading was recorded, while in the second task (TSR) the subjects had to annotate a specific relation type in each sentence  Thus, the task-specific annotation reading lead to shorter passes because the goal was merely to recognize a relation in the text, but not necessarily to process the every word in each sentence  This distinct reading behavior is shown in Figure FIGREF15, where fixations occur until the end of the sentence during normal reading, while during task-specific reading the fixations stop after the decisive words to detect a given relation type  Finally, we also analyzed the average reading times for each of the extracted eye-tracking features  The means and distributions for both tasks are shown in Figure FIGREF21  These results are in line with the recorded data in ZuCo 1 0, as well as with the features extracted in the GECO corpus BIBREF4 
Data Validation ::: EEG
As a first validation step, we extracted fixation-related potentials (FRPs), where the EEG signal during all fixations of one task are averaged  Figure FIGREF24 shows the time-series of the resulting FRPs for two electrodes (PO8 and Cz), as well as topographies of the voltage distributions across the scalp at selected points in time  The five components (for which the scalp topographies are plotted) are highly similar in the time-course of the chosen electrodes to dimigen2011coregistration as well as to ZuCo 1 0 
Moreover, these previous studies were able to show an effect of fixation duration on the resulting FRPs  To show this dependency we followed two approaches  First, for each reading task, all single-trial FRPs were ordered by fixation duration and a vertical sliding time-window was used to smooth the data BIBREF13  Figure FIGREF25 (bottom) shows the resulting plots  In line with this previous work, a first positivation can be identified at 100 ms post-fixation onset  A second positive peak is located dependent on the duration of the fixation, which can be explained by the time-jittered succeeding fixation  The second approach is based on henderson2013co in which single trial EEG segments are clustered by the duration of the current fixation  As shown in Figure FIGREF25 (top), we chose four clusters and averaged the data within each cluster to four distinct FRPs, depending on the fixation duration  Again, the same positivation peaks become apparent  Both findings are consistent with the previous work mentioned and with our findings from ZuCo 1 0 
Conclusion
We presented a new, freely available corpus of eye movement and electrical brain activity recordings during natural reading as well as during annotation  This is the first dataset that allows for the comparison between these two reading paradigms  We described the materials and experiment design in detail and conducted an extensive validation to ensure the quality of the recorded data  Since this corpus is tailored to cognitively-inspired NLP, the applications and re-use potentials of this data are extensive  The provided word-level and sentence-level eye-tracking and EEG features can be used to improve and evaluate NLP and machine learning methods, for instance, to evaluate linguistic phenomena in neural models via psycholinguistic data  In addition, because the sentences contains semantic relation labels and the annotations of all participants, it can also be widely used for relation extraction and classification  Finally, the two carefully constructed reading paradigms allow for the comparison between normal reading and reading during annotation, which can be relevant to improve the manual labelling process as well as the quality of the annotations for supervised machine learning 

ground_truth: translation probabilities, Labeled Attachment Scores (LAS)

accuracy, Labeled Attachment Scores (LAS)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How many different types of entities exist in the dataset?
Context 2:  The sentences are separated by empty newline  A sample sentence from the dataset is presented in table FIGREF13 
Dataset Statistics ::: ILPRL dataset
After much time, we received the dataset from Bal Krishna Bal, ILPRL, KU  This dataset follows standard CoNLL-2003 IOB formatBIBREF25 with POS tags  This dataset is prepared by ILPRL Lab, KU and KEIV Technologies  Few corrections like correcting the NER tags had to be made on the dataset  The statistics of both the dataset is presented in table TABREF23 
Table TABREF24 presents the total entities (PER, LOC, ORG and MISC) from both of the dataset used in our experiments  The dataset is divided into three parts with 64%, 16% and 20% of the total dataset into training set, development set and test set respectively 
Experiments
In this section, we present the details about training our neural network  The neural network architecture are implemented using PyTorch framework BIBREF26  The training is performed on a single Nvidia Tesla P100 SXM2  We first run our experiment on BiLSTM, BiLSTM-CNN, BiLSTM-CRF BiLSTM-CNN-CRF using the hyper-parameters mentioned in Table TABREF30  The training and evaluation was done on sentence-level  The RNN variants are initialized randomly from $(-\sqrt{k},\sqrt{k})$ where $k=\frac{1}{hidden\_size}$ 
First we loaded our dataset and built vocabulary using torchtext library  This eased our process of data loading using its SequenceTaggingDataset class  We trained our model with shuffled training set using Adam optimizer with hyper-parameters mentioned in table TABREF30  All our models were trained on single layer of LSTM network  We found out Adam was giving better performance and faster convergence compared to Stochastic Gradient Descent (SGD)  We chose those hyper-parameters after many ablation studies  The dropout of 0 5 is applied after LSTM layer 
For CNN, we used 30 different filters of sizes 3, 4 and 5  The embeddings of each character or grapheme involved in a given word, were passed through the pipeline of Convolution, Rectified Linear Unit and Max-Pooling  The resulting vectors were concatenated and applied dropout of 0 5 before passing into linear layer to obtain the embedding size of 30 for the given word  This resulting embedding is concatenated with word embeddings, which is again concatenated with one-hot POS vector 
Experiments ::: Tagging Scheme
Currently, for our experiments we trained our model on IO (Inside, Outside) format for both the dataset, hence the dataset does not contain any B-type annotation unlike in BIO (Beginning, Inside, Outside) scheme 
Experiments ::: Early Stopping
We used simple early stopping technique where if the validation loss does not decrease after 10 epochs, the training was stopped, else the training will run upto 100 epochs  In our experience, training usually stops around 30-50 epochs 
Experiments ::: Hyper-parameters Tuning
We ran our experiment looking for the best hyper-parameters by changing learning rate from (0,1, 0 01, 0 001, 0 0001), weight decay from [$10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$, $10^{-7}$], batch size from [1, 2, 4, 8, 16, 32, 64, 128], hidden size from [8, 16, 32, 64, 128, 256, 512 1024]  Table TABREF30 shows all other hyper-parameter used in our experiment for both of the dataset 
Experiments ::: Effect of Dropout
Figure FIGREF31 shows how we end up choosing 0 5 as dropout rate  When the dropout layer was not used, the F1 score are at the lowest  As, we slowly increase the dropout rate, the F1 score also gradually increases, however after dropout rate = 0 5, the F1 score starts falling down  Therefore, we have chosen 0 5 as dropout rate for all other experiments performed 
Evaluation
In this section, we present the details regarding evaluation and comparison of our models with other baselines 
Table TABREF25 shows the study of various embeddings and comparison among each other in OurNepali dataset  Here, raw dataset represents such dataset where post-positions are not lemmatized  We can observe that pre-trained embeddings significantly improves the score compared to randomly initialized embedding  We can deduce that Skip Gram models perform better compared CBOW models for word2vec and fasttext  Here, fastText_Pretrained represents the embedding readily available in fastText website, while other embeddings are trained on the Nepali National Corpus as mentioned in sub-section SECREF11  From this table TABREF25, we can clearly observe that model using fastText_Skip Gram embeddings outperforms all other models 
Table TABREF35 shows the model architecture comparison between all the models experimented  The features used for Stanford CRF classifier are words, letter n-grams of upto length 6, previous word and next word  This model is trained till the current function value is less than $1\mathrm {e}{-2}$  The hyper-parameters of neural network experiments are set as shown in table TABREF30  Since, word embedding of character-level and grapheme-level is random, their scores are near 
All models are evaluated using CoNLL-2003 evaluation scriptBIBREF25 to calculate entity-wise precision, recall and f1 score 
Discussion
In this paper we present that we can exploit the power of neural network to train the model to perform downstream NLP tasks like Named Entity Recognition even in Nepali language  We showed that the word vectors learned through fasttext skip gram model performs better than other word embedding because of its capability to represent sub-word and this is particularly important to capture morphological structure of words and sentences in highly inflectional like Nepali  This concept can come handy in other Devanagari languages as well because the written scripts have similar syntactical structure 
We also found out that stemming post-positions can help a lot in improving model performance because of inflectional characteristics of Nepali language  So when we separate out its inflections or morphemes, we can minimize the variations of same word which gives its root word a stronger word vector representations compared to its inflected versions 
We can clearly imply from tables TABREF23, TABREF24, and TABREF35 that we need more data to get better results because OurNepali dataset volume is almost ten times bigger compared to ILPRL dataset in terms of entities 
Conclusion and Future work
In this paper, we proposed a novel NER for Nepali language and achieved relative improvement of upto 10% and studies different factors effecting the performance of the NER for Nepali language 
We also present a neural architecture BiLSTM+CNN(grapheme-level) which turns out to be performing on par with BiLSTM+CNN(character-level) under the same configuration  We believe this will not only help Nepali language but also other languages falling under the umbrellas of Devanagari languages  Our model BiLSTM+CNN(grapheme-level) and BiLSTM+CNN(G)+POS outperforms all other model experimented in OurNepali and ILPRL dataset respectively 
Since this is the first named entity recognition research in Nepal language using neural network, there are many rooms for improvement  We believe initializing the grapheme-level embedding with fasttext embeddings might help boosting the performance, rather than randomly initializing it  In future, we plan to apply other latest techniques like BERT, ELMo and FLAIR to study its effect on low-resource language like Nepali  We also plan to improve the model using cross-lingual or multi-lingual parameter sharing techniques by jointly training with other Devanagari languages like Hindi and Bengali 
Finally, we would like to contribute our dataset to Nepali NLP community to move forward the research going on in language understanding domain  We believe there should be special committee to create and maintain such dataset for Nepali NLP and organize various competitions which would elevate the NLP research in Nepal 
Some of the future works are listed below:
Proper initialization of grapheme level embedding from fasttext embeddings 
Apply robust POS-tagger for Nepali dataset
Lemmatize the OurNepali dataset with robust and efficient lemmatizer
Improve Nepali language score with cross-lingual learning techniques
Create more dataset using Wikipedia/Wikidata framework
Acknowledgments
The authors of this paper would like to express sincere thanks to Bal Krishna Bal, Kathmandu University Professor for providing us the POS-tagged Nepali NER data 

ground_truth: OurNepali contains 3 different types of entities, ILPRL contains 4 different types of entities

three

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Did they experiment with this new dataset?
Context 4: 1$ for sentence classification based on dev (internal, fold1) scores  We use BIO tagging scheme of NER in FLC task  For CNN, we follow DBLP:conf/emnlp/Kim14 with filter-sizes of [2, 3, 4, 5, 6], 128 filters and 16 batch-size  We compute binary-F1and macro-F1 BIBREF12 in SLC and FLC, respectively on dev (internal) 
Experiments and Evaluation ::: Results: Sentence-Level Propaganda
Table TABREF10 shows the scores on dev (internal and external) for SLC task  Observe that the pre-trained embeddings (FastText or BERT) outperform TF-IDF vector representation  In row r2, we apply logistic regression classifier with BERTSentEmb that leads to improved scores over FastTextSentEmb  Subsequently, we augment the sentence vector with additional features that improves F1 on dev (external), however not dev (internal)  Next, we initialize CNN by FastTextWordEmb or BERTWordEmb and augment the last hidden layer (before classification) with BERTSentEmb and feature vectors, leading to gains in F1 for both the dev sets  Further, we fine-tune BERT and apply different thresholds in relaxing the decision boundary, where $\tau \ge 0 35$ is found optimal 
We choose the three different models in the ensemble: Logistic Regression, CNN and BERT on fold1 and subsequently an ensemble+ of r3, r6 and r12 from each fold1-5 (i e , 15 models) to obtain predictions for dev (external)  We investigate different ensemble schemes (r17-r19), where we observe that the relax-voting improves recall and therefore, the higher F1 (i e , 0 673)  In postprocess step, we check for repetition propaganda technique by computing cosine similarity between the current sentence and its preceding $w=10$ sentence vectors (i e , BERTSentEmb) in the document  If the cosine-similarity is greater than $\lambda \in \lbrace  99,  95\rbrace $, then the current sentence is labeled as propaganda due to repetition  Comparing r19 and r21, we observe a gain in recall, however an overall decrease in F1 applying postprocess 
Finally, we use the configuration of r19 on the test set  The ensemble+ of (r4, r7 r12) was analyzed after test submission  Table TABREF9 (SLC) shows that our submission is ranked at 4th position 
Experiments and Evaluation ::: Results: Fragment-Level Propaganda
Table TABREF11 shows the scores on dev (internal and external) for FLC task  Observe that the features (i e , polarity, POS and NER in row II) when introduced in LSTM-CRF improves F1  We run multi-grained LSTM-CRF without BERTSentEmb (i e , row III) and with it (i e , row IV), where the latter improves scores on dev (internal), however not on dev (external)  Finally, we perform multi-tasking with another auxiliary task of PFD  Given the scores on dev (internal and external) using different configurations (rows I-V), it is difficult to infer the optimal configuration  Thus, we choose the two best configurations (II and IV) on dev (internal) set and build an ensemble+ of predictions (discussed in section SECREF6), leading to a boost in recall and thus an improved F1 on dev (external) 
Finally, we use the ensemble+ of (II and IV) from each of the folds 1-3, i e , $|{\mathcal {M}}|=6$ models to obtain predictions on test  Table TABREF9 (FLC) shows that our submission is ranked at 3rd position 
Conclusion and Future Work
Our system (Team: MIC-CIS) explores different neural architectures (CNN, BERT and LSTM-CRF) with linguistic, layout and topical features to address the tasks of fine-grained propaganda detection  We have demonstrated gains in performance due to the features, ensemble schemes, multi-tasking and multi-granularity architectures  Compared to the other participating systems, our submissions are ranked 3rd and 4th in FLC and SLC tasks, respectively 
In future, we would like to enrich BERT models with linguistic, layout and topical features during their fine-tuning  Further, we would also be interested in understanding and analyzing the neural network learning, i e , extracting salient fragments (or key-phrases) in the sentence that generate propaganda, similar to pankajgupta:2018LISA in order to promote explainable AI 

ground_truth: No