prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which languages are similar to each other?
Context 1: Introduction
Accurate language identification (LID) is the first step in many natural language processing and machine comprehension pipelines  If the language of a piece of text is known then the appropriate downstream models like parts of speech taggers and language models can be applied as required 
LID is further also an important step in harvesting scarce language resources  Harvested data can be used to bootstrap more accurate LID models and in doing so continually improve the quality of the harvested data  Availability of data is still one of the big roadblocks for applying data driven approaches like supervised machine learning in developing countries 
Having 11 official languages of South Africa has lead to initiatives (discussed in the next section) that have had positive effect on the availability of language resources for research  However, many of the South African languages are still under resourced from the point of view of building data driven models for machine comprehension and process automation 
Table TABREF2 shows the percentages of first language speakers for each of the official languages of South Africa  These are four conjunctively written Nguni languages (zul, xho, nbl, ssw), Afrikaans (afr) and English (eng), three disjunctively written Sotho languages (nso, sot, tsn), as well as tshiVenda (ven) and Xitsonga (tso)  The Nguni languages are similar to each other and harder to distinguish  The same is true of the Sotho languages 
This paper presents a hierarchical naive Bayesian and lexicon based classifier for LID of short pieces of text of 15-20 characters long  The algorithm is evaluated against recent approaches using existing test sets from previous works on South African languages as well as the Discriminating between Similar Languages (DSL) 2015 and 2017 shared tasks 
Section SECREF2 reviews existing works on the topic and summarises the remaining research problems  Section SECREF3 of the paper discusses the proposed algorithm and Section SECREF4 presents comparative results 
Related Works
The focus of this section is on recently published datasets and LID research applicable to the South African context  An in depth survey of algorithms, features, datasets, shared tasks and evaluation methods may be found in BIBREF0 
The datasets for the DSL 2015 & DSL 2017 shared tasks BIBREF1 are often used in LID benchmarks and also available on Kaggle   The DSL datasets, like other LID datasets, consists of text sentences labelled by language  The 2017 dataset, for example, contains 14 languages over 6 language groups with 18000 training samples and 1000 testing samples per language 
The recently published JW300 parallel corpus BIBREF2 covers over 300 languages with around 100 thousand parallel sentences per language pair on average  In South Africa, a multilingual corpus of academic texts produced by university students with different mother tongues is being developed BIBREF3  The WiLI-2018 benchmark dataset BIBREF4 for monolingual written natural language identification includes around 1000 paragraphs of 235 languages  A possibly useful link can also be made BIBREF5 between Native Language Identification (NLI) (determining the native language of the author of a text) and Language Variety Identification (LVI) (classification of different varieties of a single language) which opens up more datasets  The Leipzig Corpora Collection BIBREF6, the Universal Declaration of Human Rights and Tatoeba are also often used sources of data 
The NCHLT text corpora BIBREF7 is likely a good starting point for a shared LID task dataset for the South African languages BIBREF8  The NCHLT text corpora contains enough data to have 3500 training samples and 600 testing samples of 300+ character sentences per language  Researchers have recently started applying existing algorithms for tasks like neural machine translation in earnest to such South African language datasets BIBREF9 
Existing NLP datasets, models and services BIBREF10 are available for South African languages  These include an LID algorithm BIBREF11 that uses a character level n-gram language model  Multiple papers have shown that 'shallow' naive Bayes classifiers BIBREF12, BIBREF8, BIBREF13, BIBREF14, SVMs BIBREF15 and similar models work very well for doing LID  The DSL 2017 paper BIBREF1, for example, gives an overview of the solutions of all of the teams that competed on the shared task and the winning approach BIBREF16 used an SVM with character n-gram, parts of speech tag features and some other engineered features  The winning approach for DSL 2015 used an ensemble naive Bayes classifier  The fasttext classifier BIBREF17 is perhaps one of the best known efficient 'shallow' text classifiers that have been used for LID  
Multiple papers have proposed hierarchical stacked classifiers (including lexicons) that would for example first classify a piece of text by language group and then by exact language BIBREF18, BIBREF19, BIBREF8, BIBREF0  Some work has also been done on classifying surnames between Tshivenda, Xitsonga and Sepedi BIBREF20  Additionally, data augmentation BIBREF21 and adversarial training BIBREF22 approaches are potentially very useful to reduce the requirement for data 
Researchers have investigated deeper LID models like bidirectional recurrent neural networks BIBREF23 or ensembles of recurrent neural networks BIBREF24  The latter is reported to achieve 95 12% in the DSL 2015 shared task  In these models text features can include character and word n-grams as well as informative character and word-level features learnt BIBREF25 from the training data  The neural methods seem to work well in tasks where more training data is available 
In summary, LID of short texts, informal styles and similar languages remains a difficult problem which is actively being researched  Increased confusion can in general be expected between shorter pieces of text and languages that are more closely related  Shallow methods still seem to work well compared to deeper models for LID  Other remaining research opportunities seem to be data harvesting, building standardised datasets and creating shared tasks for South Africa and Africa  Support for language codes that include more languages seems to be growing and discoverability of research is improving with more survey papers coming out  Paywalls also seem to no longer be a problem; the references used in this paper was either openly published or available as preprint papers 
Methodology
The proposed LID algorithm builds on the work in BIBREF8 and BIBREF26  We apply a naive Bayesian classifier with character (2, 4 & 6)-grams, word unigram and word bigram features with a hierarchical lexicon based classifier 
The naive Bayesian classifier is trained to predict the specific language label of a piece of text, but used to first classify text as belonging to either the Nguni family, the Sotho family, English, Afrikaans, Xitsonga or Tshivenda  The scikit-learn multinomial naive Bayes classifier is used for the implementation with an alpha smoothing value of 0 01 and hashed text features 
The lexicon based classifier is then used to predict the specific language within a language group  For the South African languages this is done for the Nguni and Sotho groups  If the lexicon prediction of the specific language has high confidence then its result is used as the final label else the naive Bayesian classifier's specific language prediction is used as the final result  The lexicon is built over all the data and therefore includes the vocabulary from both the training and testing sets 
The lexicon based classifier is designed to trade higher precision for lower recall  The proposed implementation is considered confident if the number of words from the winning language is at least one more than the number of words considered to be from the language scored in second place 
The stacked classifier is tested against three public LID implementations BIBREF17, BIBREF23, BIBREF8  The LID implementation described in BIBREF17 is available on GitHub and is trained and tested according to a post on the fasttext blog  Character (5-6)-gram features with 16 dimensional vectors worked the best  The implementation discussed in BIBREF23 is available from https://github com/tomkocmi/LanideNN  Following the instructions for an OSX pip install of an old r0 8 release of TensorFlow, the LanideNN code could be executed in Python 3 7 4  Settings were left at their defaults and a learning rate of 0 001 was used followed by a refinement with learning rate of 0 0001  Only one code modification was applied to return the results from a method that previously just printed to screen  The LID algorithm described in BIBREF8 is also available on GitHub 
The stacked classifier is also tested against the results reported for four other algorithms BIBREF16, BIBREF26, BIBREF24, BIBREF15  All the comparisons are done using the NCHLT BIBREF7, DSL 2015 BIBREF19 and DSL 2017 BIBREF1 datasets discussed in Section SECREF2 
Results and Analysis
The average classification accuracy results are summarised in Table TABREF9  The accuracies reported are for classifying a piece of text by its specific language label  Classifying text only by language group or family is a much easier task as reported in BIBREF8 
Different variations of the proposed classifier were evaluated  A single NB classifier (NB), a stack of two NB classifiers (NB+NB), a stack of a NB classifier and lexicon (NB+Lex) and a lexicon (Lex) by itself  A lexicon with a 50% training token dropout is also listed to show the impact of the lexicon support on the accuracy 
From the results it seems that the DSL 2017 task might be harder than the DSL 2015 and NCHLT tasks  Also, the results for the implementation discussed in BIBREF23 might seem low, but the results reported in that paper is generated on longer pieces of text so lower scores on the shorter pieces of text derived from the NCHLT corpora is expected 
The accuracy of the proposed algorithm seems to be dependent on the support of the lexicon  Without a good lexicon a non-stacked naive Bayesian classifier might even perform better 
The execution performance of some of the LID implementations are shown in Table TABREF10  Results were generated on an early 2015 13-inch Retina MacBook Pro with a 2 9 GHz CPU (Turbo Boosted to 3 4 GHz) and 8GB RAM  The C++ implementation in BIBREF17 is the fastest  The implementation in BIBREF8 makes use of un-hashed feature representations which causes it to be slower than the proposed sklearn implementation  The execution performance of BIBREF23 might improve by a factor of five to ten when executed on a GPU 
Conclusion
LID of short texts, informal styles and similar languages remains a difficult problem which is actively being researched  The proposed algorithm was evaluated on three existing datasets and compared to the implementations of three public LID implementations as well as to reported results of four other algorithms  It performed well relative to the other methods beating their results  However, the performance is dependent on the support of the lexicon 
We would like to investigate the value of a lexicon in a production system and how to possibly maintain it using self-supervised learning  We are investigating the application of deeper language models some of which have been used in more recent DSL shared tasks  We would also like to investigate data augmentation strategies to reduce the amount of training data that is required 
Further research opportunities include data harvesting, building standardised datasets and shared tasks for South Africa as well as the rest of Africa  In general, the support for language codes that include more languages seems to be growing, discoverability of research is improving and paywalls seem to no longer be a big problem in getting access to published research 
Context 2: Introduction
Named Entity Recognition (NER) is a foremost NLP task to label each atomic elements of a sentence into specific categories like "PERSON", "LOCATION", "ORGANIZATION" and othersBIBREF0  There has been an extensive NER research on English, German, Dutch and Spanish language BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, and notable research on low resource South Asian languages like HindiBIBREF6, IndonesianBIBREF7 and other Indian languages (Kannada, Malayalam, Tamil and Telugu)BIBREF8  However, there has been no study on developing neural NER for Nepali language  In this paper, we propose a neural based Nepali NER using latest state-of-the-art architecture based on grapheme-level which doesn't require any hand-crafted features and no data pre-processing 
Recent neural architecture like BIBREF1 is used to relax the need to hand-craft the features and need to use part-of-speech tag to determine the category of the entity  However, this architecture have been studied for languages like English, and German and not been applied to languages like Nepali which is a low resource language i e limited data set to train the model  Traditional methods like Hidden Markov Model (HMM) with rule based approachesBIBREF9,BIBREF10, and Support Vector Machine (SVM) with manual feature-engineeringBIBREF11 have been applied but they perform poor compared to neural  However, there has been no research in Nepali NER using neural network  Therefore, we created the named entity annotated dataset partly with the help of Dataturk to train a neural model  The texts used for this dataset are collected from various daily news sources from Nepal around the year 2015-2016 
Following are our contributions:
We present a novel Named Entity Recognizer (NER) for Nepali language  To best of our knowledge we are the first to propose neural based Nepali NER 
As there are not good quality dataset to train NER we release a dataset to support future research
We perform empirical evaluation of our model with state-of-the-art models with relative improvement of upto 10%
In this paper, we present works similar to ours in Section SECREF2  We describe our approach and dataset statistics in Section SECREF3 and SECREF4, followed by our experiments, evaluation and discussion in Section SECREF5, SECREF6, and SECREF7  We conclude with our observations in Section SECREF8 
To facilitate further research our code and dataset will be made available at github com/link-yet-to-be-updated
Related Work
There has been a handful of research on Nepali NER task based on approaches like Support Vector Machine and gazetteer listBIBREF11 and Hidden Markov Model and gazetteer listBIBREF9,BIBREF10 
BIBREF11 uses SVM along with features like first word, word length, digit features and gazetteer (person, organization, location, middle name, verb, designation and others)  It uses one vs rest classification model to classify each word into different entity classes  However, it does not the take context word into account while training the model  Similarly, BIBREF9 and BIBREF10 uses Hidden Markov Model with n-gram technique for extracting POS-tags  POS-tags with common noun, proper noun or combination of both are combined together, then uses gazetteer list as look-up table to identify the named entities 
Researchers have shown that the neural networks like CNNBIBREF12, RNNBIBREF13, LSTMBIBREF14, GRUBIBREF15 can capture the semantic knowledge of language better with the help of pre-trained embbeddings like word2vecBIBREF16, gloveBIBREF17 or fasttextBIBREF18 
Similar approaches has been applied to many South Asian languages like HindiBIBREF6, IndonesianBIBREF7, BengaliBIBREF19 and In this paper, we present the neural network architecture for NER task in Nepali language, which doesn't require any manual feature engineering nor any data pre-processing during training  First we are comparing BiLSTMBIBREF14, BiLSTM+CNNBIBREF20, BiLSTM+CRFBIBREF1, BiLSTM+CNN+CRFBIBREF2 models with CNN modelBIBREF0 and Stanford CRF modelBIBREF21  Secondly, we show the comparison between models trained on general word embeddings, word embedding + character-level embedding, word embedding + part-of-speech(POS) one-hot encoding and word embedding + grapheme clustered or sub-word embeddingBIBREF22  The experiments were performed on the dataset that we created and on the dataset received from ILPRL lab  Our extensive study shows that augmenting word embedding with character or grapheme-level representation and POS one-hot encoding vector yields better results compared to using general word embedding alone 
Approach
In this section, we describe our approach in building our model  This model is partly inspired from multiple models BIBREF20,BIBREF1, andBIBREF2
Approach ::: Bidirectional LSTM
We used Bi-directional LSTM to capture the word representation in forward as well as reverse direction of a sentence  Generally, LSTMs take inputs from left (past) of the sentence and computes the hidden state  However, it is proven beneficialBIBREF23 to use bi-directional LSTM, where, hidden states are computed based from right (future) of sentence and both of these hidden states are concatenated to produce the final output as $h_t$=[$\overrightarrow{h_t}$;$\overleftarrow{h_t}$], where $\overrightarrow{h_t}$, $\overleftarrow{h_t}$ = hidden state computed in forward and backward direction respectively 
Approach ::: Features ::: Word embeddings
We have used Word2Vec BIBREF16, GloVe BIBREF17 and FastText BIBREF18 word vectors of 300 dimensions  These vectors were trained on the corpus obtained from Nepali National Corpus  This pre-lemmatized corpus consists of 14 million words from books, web-texts and news papers  This corpus was mixed with the texts from the dataset before training CBOW and skip-gram version of word2vec using gensim libraryBIBREF24  This trained model consists of vectors for 72782 unique words 
Light pre-processing was performed on the corpus before training it  For example, invalid characters or characters other than Devanagari were removed but punctuation and numbers were not removed  We set the window context at 10 and the rare words whose count is below 5 are dropped  These word embeddings were not frozen during the training session because fine-tuning word embedding help achieve better performance compared to frozen oneBIBREF20 
We have used fasttext embeddings in particular because of its sub-word representation ability, which is very useful in highly inflectional language as shown in Table TABREF25  We have trained the word embedding in such a way that the sub-word size remains between 1 and 4  We particularly chose this size because in Nepali language a single letter can also be a word, for example e, t, C, r, l, n, u and a single character (grapheme) or sub-word can be formed after mixture of dependent vowel signs with consonant letters for example, C + O + = CO, here three different consonant letters form a single sub-word 
The two-dimensional visualization of an example word npAl is shown in FIGREF14  Principal Component Analysis (PCA) technique was used to generate this visualization which helps use to analyze the nearest neighbor words of a given sample word  84 and 104 nearest neighbors were observed using word2vec and fasttext embedding respectively on the same corpus 
Approach ::: Features ::: Character-level embeddings
BIBREF20 and BIBREF2 successfully presented that the character-level embeddings, extracted using CNN, when combined with word embeddings enhances the NER model performance significantly, as it is able to capture morphological features of a word  Figure FIGREF7 shows the grapheme-level CNN used in our model, where inputs to CNN are graphemes  Character-level CNN is also built in similar fashion, except the inputs are characters  Grapheme or Character -level embeddings are randomly initialized from [0,1] with real values with uniform distribution of dimension 30 
Approach ::: Features ::: Grapheme-level embeddings
Grapheme is atomic meaningful unit in writing system of any languages  Since, Nepali language is highly morphologically inflectional, we compared grapheme-level representation with character-level representation to evaluate its effect  For example, in character-level embedding, each character of a word npAl results into n + + p + A + l has its own embedding  However, in grapheme level, a word npAl is clustered into graphemes, resulting into n + pA + l  Here, each grapheme has its own embedding  This grapheme-level embedding results good scores on par with character-level embedding in highly inflectional languages like Nepali, because graphemes also capture syntactic information similar to characters  We created grapheme clusters using uniseg package which is helpful in unicode text segmentations 
Approach ::: Features ::: Part-of-speech (POS) one hot encoding
We created one-hot encoded vector of POS tags and then concatenated with pre-trained word embeddings before passing it to BiLSTM network  A sample of data is shown in figure FIGREF13 
Dataset Statistics ::: OurNepali dataset
Since, we there was no publicly available standard Nepali NER dataset and did not receive any dataset from the previous researchers, we had to create our own dataset  This dataset contains the sentences collected from daily newspaper of the year 2015-2016  This dataset has three major classes Person (PER), Location (LOC) and Organization (ORG)  Pre-processing was performed on the text before creation of the dataset, for example all punctuations and numbers besides ',', '-', '|' and ' ' were removed  Currently, the dataset is in standard CoNLL-2003 IO formatBIBREF25 
Since, this dataset is not lemmatized originally, we lemmatized only the post-positions like Ek, kO, l, mA, m, my, jF, sg, aEG which are just the few examples among 299 post positions in Nepali language  We obtained these post-positions from sanjaalcorps and added few more to match our dataset  We will be releasing this list in our github repository  We found out that lemmatizing the post-positions boosted the F1 score by almost 10% 
In order to label our dataset with POS-tags, we first created POS annotated dataset of 6946 sentences and 16225 unique words extracted from POS-tagged Nepali National Corpus and trained a BiLSTM model with 95 14% accuracy which was used to create POS-tags for our dataset 
The dataset released in our github repository contains each word in newline with space separated POS-tags and Entity-tags
Context 3: Introduction
Understanding the emotions expressed in a text or message is of high relevance nowadays  Companies are interested in this to get an understanding of the sentiment of their current customers regarding their products and the sentiment of their potential customers to attract new ones  Moreover, changes in a product or a company may also affect the sentiment of a customer  However, the intensity of an emotion is crucial in determining the urgency and importance of that sentiment  If someone is only slightly happy about a product, is a customer willing to buy it again? Conversely, if someone is very angry about customer service, his or her complaint might be given priority over somewhat milder complaints 
BIBREF0 present four tasks in which systems have to automatically determine the intensity of emotions (EI) or the intensity of the sentiment (Valence) of tweets in the languages English, Arabic, and Spanish  The goal is to either predict a continuous regression (reg) value or to do ordinal classification (oc) based on a number of predefined categories  The EI tasks have separate training sets for four different emotions: anger, fear, joy and sadness  Due to the large number of subtasks and the fact that this language does not have many resources readily available, we only focus on the Spanish subtasks  Our work makes the following contributions:
Our submissions ranked second (EI-Reg), second (EI-Oc), fourth (V-Reg) and fifth (V-Oc), demonstrating that the proposed method is accurate in automatically determining the intensity of emotions and sentiment of Spanish tweets  This paper will first focus on the datasets, the data generation procedure, and the techniques and tools used  Then we present the results in detail, after which we perform a small error analysis on the largest mistakes our model made  We conclude with some possible ideas for future work 
Data
For each task, the training data that was made available by the organizers is used, which is a selection of tweets with for each tweet a label describing the intensity of the emotion or sentiment BIBREF1   Links and usernames were replaced by the general tokens URL and @username, after which the tweets were tokenized by using TweetTokenizer  All text was lowercased  In a post-processing step, it was ensured that each emoji is tokenized as a single token 
Word Embeddings
To be able to train word embeddings, Spanish tweets were scraped between November 8, 2017 and January 12, 2018  We chose to create our own embeddings instead of using pre-trained embeddings, because this way the embeddings would resemble the provided data set: both are based on Twitter data  Added to this set was the Affect in Tweets Distant Supervision Corpus (DISC) made available by the organizers BIBREF0 and a set of 4 1 million tweets from 2015, obtained from BIBREF2   After removing duplicate tweets and tweets with fewer than ten tokens, this resulted in a set of 58 7 million tweets, containing 1 1 billion tokens  The tweets were preprocessed using the method described in Section SECREF6   The word embeddings were created using word2vec in the gensim library BIBREF3 , using CBOW, a window size of 40 and a minimum count of 5  The feature vectors for each tweet were then created by using the AffectiveTweets WEKA package BIBREF4  
Translating Lexicons
Most lexical resources for sentiment analysis are in English  To still be able to benefit from these sources, the lexicons in the AffectiveTweets package were translated to Spanish, using the machine translation platform Apertium BIBREF5  
All lexicons from the AffectiveTweets package were translated, except for SentiStrength  Instead of translating this lexicon, the English version was replaced by the Spanish variant made available by BIBREF6  
For each subtask, the optimal combination of lexicons was determined  This was done by first calculating the benefits of adding each lexicon individually, after which only beneficial lexicons were added until the score did not increase anymore (e g  after adding the best four lexicons the fifth one did not help anymore, so only four were added)  The tests were performed using a default SVM model, with the set of word embeddings described in the previous section  Each subtask thus uses a different set of lexicons (see Table TABREF1 for an overview of the lexicons used in our final ensemble)  For each subtask, this resulted in a (modest) increase on the development set, between 0 01 and 0 05 
Translating Data
The training set provided by BIBREF0 is not very large, so it was interesting to find a way to augment the training set  A possible method is to simply translate the datasets into other languages, leaving the labels intact  Since the present study focuses on Spanish tweets, all tweets from the English datasets were translated into Spanish  This new set of “Spanish” data was then added to our original training set  Again, the machine translation platform Apertium BIBREF5 was used for the translation of the datasets

ground_truth: Nguni languages (zul, xho, nbl, ssw), Sotho languages (nso, sot, tsn)

The Nguni languages are similar to each other, The same is true of the Sotho languages

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the size of the dataset?
Context 2:  The sentences are separated by empty newline  A sample sentence from the dataset is presented in table FIGREF13 
Dataset Statistics ::: ILPRL dataset
After much time, we received the dataset from Bal Krishna Bal, ILPRL, KU  This dataset follows standard CoNLL-2003 IOB formatBIBREF25 with POS tags  This dataset is prepared by ILPRL Lab, KU and KEIV Technologies  Few corrections like correcting the NER tags had to be made on the dataset  The statistics of both the dataset is presented in table TABREF23 
Table TABREF24 presents the total entities (PER, LOC, ORG and MISC) from both of the dataset used in our experiments  The dataset is divided into three parts with 64%, 16% and 20% of the total dataset into training set, development set and test set respectively 
Experiments
In this section, we present the details about training our neural network  The neural network architecture are implemented using PyTorch framework BIBREF26  The training is performed on a single Nvidia Tesla P100 SXM2  We first run our experiment on BiLSTM, BiLSTM-CNN, BiLSTM-CRF BiLSTM-CNN-CRF using the hyper-parameters mentioned in Table TABREF30  The training and evaluation was done on sentence-level  The RNN variants are initialized randomly from $(-\sqrt{k},\sqrt{k})$ where $k=\frac{1}{hidden\_size}$ 
First we loaded our dataset and built vocabulary using torchtext library  This eased our process of data loading using its SequenceTaggingDataset class  We trained our model with shuffled training set using Adam optimizer with hyper-parameters mentioned in table TABREF30  All our models were trained on single layer of LSTM network  We found out Adam was giving better performance and faster convergence compared to Stochastic Gradient Descent (SGD)  We chose those hyper-parameters after many ablation studies  The dropout of 0 5 is applied after LSTM layer 
For CNN, we used 30 different filters of sizes 3, 4 and 5  The embeddings of each character or grapheme involved in a given word, were passed through the pipeline of Convolution, Rectified Linear Unit and Max-Pooling  The resulting vectors were concatenated and applied dropout of 0 5 before passing into linear layer to obtain the embedding size of 30 for the given word  This resulting embedding is concatenated with word embeddings, which is again concatenated with one-hot POS vector 
Experiments ::: Tagging Scheme
Currently, for our experiments we trained our model on IO (Inside, Outside) format for both the dataset, hence the dataset does not contain any B-type annotation unlike in BIO (Beginning, Inside, Outside) scheme 
Experiments ::: Early Stopping
We used simple early stopping technique where if the validation loss does not decrease after 10 epochs, the training was stopped, else the training will run upto 100 epochs  In our experience, training usually stops around 30-50 epochs 
Experiments ::: Hyper-parameters Tuning
We ran our experiment looking for the best hyper-parameters by changing learning rate from (0,1, 0 01, 0 001, 0 0001), weight decay from [$10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$, $10^{-7}$], batch size from [1, 2, 4, 8, 16, 32, 64, 128], hidden size from [8, 16, 32, 64, 128, 256, 512 1024]  Table TABREF30 shows all other hyper-parameter used in our experiment for both of the dataset 
Experiments ::: Effect of Dropout
Figure FIGREF31 shows how we end up choosing 0 5 as dropout rate  When the dropout layer was not used, the F1 score are at the lowest  As, we slowly increase the dropout rate, the F1 score also gradually increases, however after dropout rate = 0 5, the F1 score starts falling down  Therefore, we have chosen 0 5 as dropout rate for all other experiments performed 
Evaluation
In this section, we present the details regarding evaluation and comparison of our models with other baselines 
Table TABREF25 shows the study of various embeddings and comparison among each other in OurNepali dataset  Here, raw dataset represents such dataset where post-positions are not lemmatized  We can observe that pre-trained embeddings significantly improves the score compared to randomly initialized embedding  We can deduce that Skip Gram models perform better compared CBOW models for word2vec and fasttext  Here, fastText_Pretrained represents the embedding readily available in fastText website, while other embeddings are trained on the Nepali National Corpus as mentioned in sub-section SECREF11  From this table TABREF25, we can clearly observe that model using fastText_Skip Gram embeddings outperforms all other models 
Table TABREF35 shows the model architecture comparison between all the models experimented  The features used for Stanford CRF classifier are words, letter n-grams of upto length 6, previous word and next word  This model is trained till the current function value is less than $1\mathrm {e}{-2}$  The hyper-parameters of neural network experiments are set as shown in table TABREF30  Since, word embedding of character-level and grapheme-level is random, their scores are near 
All models are evaluated using CoNLL-2003 evaluation scriptBIBREF25 to calculate entity-wise precision, recall and f1 score 
Discussion
In this paper we present that we can exploit the power of neural network to train the model to perform downstream NLP tasks like Named Entity Recognition even in Nepali language  We showed that the word vectors learned through fasttext skip gram model performs better than other word embedding because of its capability to represent sub-word and this is particularly important to capture morphological structure of words and sentences in highly inflectional like Nepali  This concept can come handy in other Devanagari languages as well because the written scripts have similar syntactical structure 
We also found out that stemming post-positions can help a lot in improving model performance because of inflectional characteristics of Nepali language  So when we separate out its inflections or morphemes, we can minimize the variations of same word which gives its root word a stronger word vector representations compared to its inflected versions 
We can clearly imply from tables TABREF23, TABREF24, and TABREF35 that we need more data to get better results because OurNepali dataset volume is almost ten times bigger compared to ILPRL dataset in terms of entities 
Conclusion and Future work
In this paper, we proposed a novel NER for Nepali language and achieved relative improvement of upto 10% and studies different factors effecting the performance of the NER for Nepali language 
We also present a neural architecture BiLSTM+CNN(grapheme-level) which turns out to be performing on par with BiLSTM+CNN(character-level) under the same configuration  We believe this will not only help Nepali language but also other languages falling under the umbrellas of Devanagari languages  Our model BiLSTM+CNN(grapheme-level) and BiLSTM+CNN(G)+POS outperforms all other model experimented in OurNepali and ILPRL dataset respectively 
Since this is the first named entity recognition research in Nepal language using neural network, there are many rooms for improvement  We believe initializing the grapheme-level embedding with fasttext embeddings might help boosting the performance, rather than randomly initializing it  In future, we plan to apply other latest techniques like BERT, ELMo and FLAIR to study its effect on low-resource language like Nepali  We also plan to improve the model using cross-lingual or multi-lingual parameter sharing techniques by jointly training with other Devanagari languages like Hindi and Bengali 
Finally, we would like to contribute our dataset to Nepali NLP community to move forward the research going on in language understanding domain  We believe there should be special committee to create and maintain such dataset for Nepali NLP and organize various competitions which would elevate the NLP research in Nepal 
Some of the future works are listed below:
Proper initialization of grapheme level embedding from fasttext embeddings 
Apply robust POS-tagger for Nepali dataset
Lemmatize the OurNepali dataset with robust and efficient lemmatizer
Improve Nepali language score with cross-lingual learning techniques
Create more dataset using Wikipedia/Wikidata framework
Acknowledgments
The authors of this paper would like to express sincere thanks to Bal Krishna Bal, Kathmandu University Professor for providing us the POS-tagged Nepali NER data 
Context 3: 
Algorithms Used
Three types of models were used in our system, a feed-forward neural network, an LSTM network and an SVM regressor  The neural nets were inspired by the work of Prayas BIBREF7 in the previous shared task  Different regression algorithms (e g  AdaBoost, XGBoost) were also tried due to the success of SeerNet BIBREF8 , but our study was not able to reproduce their results for Spanish 
For both the LSTM network and the feed-forward network, a parameter search was done for the number of layers, the number of nodes and dropout used  This was done for each subtask, i e  different tasks can have a different number of layers  All models were implemented using Keras BIBREF9   After the best parameter settings were found, the results of 10 system runs to produce our predictions were averaged (note that this is different from averaging our different type of models in Section SECREF16 )  For the SVM (implemented in scikit-learn BIBREF10 ), the RBF kernel was used and a parameter search was conducted for epsilon  Detailed parameter settings for each subtask are shown in Table TABREF12   Each parameter search was performed using 10-fold cross validation, as to not overfit on the development set 
Semi-supervised Learning
One of the aims of this study was to see if using semi-supervised learning is beneficial for emotion intensity tasks  For this purpose, the DISC BIBREF0 corpus was used  This corpus was created by querying certain emotion-related words, which makes it very suitable as a semi-supervised corpus  However, the specific emotion the tweet belonged to was not made public  Therefore, a method was applied to automatically assign the tweets to an emotion by comparing our scraped tweets to this new data set 
First, in an attempt to obtain the query-terms, we selected the 100 words which occurred most frequently in the DISC corpus, in comparison with their frequencies in our own scraped tweets corpus  Words that were clearly not indicators of emotion were removed  The rest was annotated per emotion or removed if it was unclear to which emotion the word belonged  This allowed us to create silver datasets per emotion, assigning tweets to an emotion if an annotated emotion-word occurred in the tweet 
Our semi-supervised approach is quite straightforward: first a model is trained on the training set and then this model is used to predict the labels of the silver data  This silver data is then simply added to our training set, after which the model is retrained  However, an extra step is applied to ensure that the silver data is of reasonable quality  Instead of training a single model initially, ten different models were trained which predict the labels of the silver instances  If the highest and lowest prediction do not differ more than a certain threshold the silver instance is maintained, otherwise it is discarded 
This results in two parameters that could be optimized: the threshold and the number of silver instances that would be added  This method can be applied to both the LSTM and feed-forward networks that were used  An overview of the characteristics of our data set with the final parameter settings is shown in Table TABREF14   Usually, only a small subset of data was added to our training set, meaning that most of the silver data is not used in the experiments  Note that since only the emotions were annotated, this method is only applicable to the EI tasks 
Ensembling
To boost performance, the SVM, LSTM, and feed-forward models were combined into an ensemble  For both the LSTM and feed-forward approach, three different models were trained  The first model was trained on the training data (regular), the second model was trained on both the training and translated training data (translated) and the third one was trained on both the training data and the semi-supervised data (silver)  Due to the nature of the SVM algorithm, semi-supervised learning does not help, so only the regular and translated model were trained in this case  This results in 8 different models per subtask  Note that for the valence tasks no silver training data was obtained, meaning that for those tasks the semi-supervised models could not be used 
Per task, the LSTM and feed-forward model's predictions were averaged over 10 prediction runs  Subsequently, the predictions of all individual models were combined into an average  Finally, models were removed from the ensemble in a stepwise manner if the removal increased the average score  This was done based on their original scores, i e  starting out by trying to remove the worst individual model and working our way up to the best model

ground_truth: Dataset contains 3606 total sentences and 79087 total entities.

ILPRL contains 548 sentences, OurNepali contains 3606 sentences

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What were the scores of their system?
Context 3:  We only consider it an increase in score if the difference is larger than 0 002 (i e  the difference between 0 716 and 0 718)  If at some point the score does not increase and we are therefore unable to remove a model, the process is stopped and our best ensemble of models has been found  This process uses the scores on the development set of different combinations of models  Note that this means that the ensembles for different subtasks can contain different sets of models  The final model selections can be found in Table TABREF17  
Results and Discussion
Table TABREF18 shows the results on the development set of all individuals models, distinguishing the three types of training: regular (r), translated (t) and semi-supervised (s)  In Tables TABREF17 and TABREF18 , the letter behind each model (e g  SVM-r, LSTM-r) corresponds to the type of training used  Comparing the regular and translated columns for the three algorithms, it shows that in 22 out of 30 cases, using translated instances as extra training data resulted in an improvement  For the semi-supervised learning approach, an improvement is found in 15 out of 16 cases  Moreover, our best individual model for each subtask (bolded scores in Table TABREF18 ) is always either a translated or semi-supervised model  Table TABREF18 also shows that, in general, our feed-forward network obtained the best results, having the highest F-score for 8 out of 10 subtasks 
However, Table TABREF19 shows that these scores can still be improved by averaging or ensembling the individual models  On the dev set, averaging our 8 individual models results in a better score for 8 out of 10 subtasks, while creating an ensemble beats all of the individual models as well as the average for each subtask  On the test set, however, only a small increase in score (if any) is found for stepwise ensembling, compared to averaging  Even though the results do not get worse, we cannot conclude that stepwise ensembling is a better method than simply averaging 
Our official scores (column Ens Test in Table TABREF19 ) have placed us second (EI-Reg, EI-Oc), fourth (V-Reg) and fifth (V-Oc) on the SemEval AIT-2018 leaderboard  However, it is evident that the results obtained on the test set are not always in line with those achieved on the development set  Especially on the anger subtask for both EI-Reg and EI-Oc, the scores are considerably lower on the test set in comparison with the results on the development set  Therefore, a small error analysis was performed on the instances where our final model made the largest errors 
Error Analysis
Due to some large differences between our results on the dev and test set of this task, we performed a small error analysis in order to see what caused these differences  For EI-Reg-anger, the gold labels were compared to our own predictions, and we manually checked 50 instances for which our system made the largest errors 
Some examples that were indicative of the shortcomings of our system are shown in Table TABREF20   First of all, our system did not take into account capitalization  The implications of this are shown in the first sentence, where capitalization intensifies the emotion used in the sentence  In the second sentence, the name Imperator Furiosa is not understood  Since our texts were lowercased, our system was unable to capture the named entity and thought the sentence was about an angry emperor instead  In the third sentence, our system fails to capture that when you are so angry that it makes you laugh, it results in a reduced intensity of the angriness  Finally, in the fourth sentence, it is the figurative language me infla la vena (it inflates my vein) that the system is not able to understand 
The first two error-categories might be solved by including smart features regarding capitalization and named entity recognition  However, the last two categories are problems of natural language understanding and will be very difficult to fix 
Conclusion
To conclude, the present study described our submission for the Semeval 2018 Shared Task on Affect in Tweets  We participated in four Spanish subtasks and our submissions ranked second, second, fourth and fifth place  Our study aimed to investigate whether the automatic generation of additional training data through translation and semi-supervised learning, as well as the creation of stepwise ensembles, increase the performance of our Spanish-language models  Strong support was found for the translation and semi-supervised learning approaches; our best models for all subtasks use either one of these approaches  These results suggest that both of these additional data resources are beneficial when determining emotion intensity (for Spanish)  However, the creation of a stepwise ensemble from the best models did not result in better performance compared to simply averaging the models  In addition, some signs of overfitting on the dev set were found  In future work, we would like to apply the methods (translation and semi-supervised learning) used on Spanish on other low-resource languages and potentially also on other tasks 

ground_truth: column Ens Test in Table TABREF19