prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How many different types of entities exist in the dataset?
Context 1: Introduction
Named Entity Recognition (NER) is a foremost NLP task to label each atomic elements of a sentence into specific categories like "PERSON", "LOCATION", "ORGANIZATION" and othersBIBREF0  There has been an extensive NER research on English, German, Dutch and Spanish language BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, and notable research on low resource South Asian languages like HindiBIBREF6, IndonesianBIBREF7 and other Indian languages (Kannada, Malayalam, Tamil and Telugu)BIBREF8  However, there has been no study on developing neural NER for Nepali language  In this paper, we propose a neural based Nepali NER using latest state-of-the-art architecture based on grapheme-level which doesn't require any hand-crafted features and no data pre-processing 
Recent neural architecture like BIBREF1 is used to relax the need to hand-craft the features and need to use part-of-speech tag to determine the category of the entity  However, this architecture have been studied for languages like English, and German and not been applied to languages like Nepali which is a low resource language i e limited data set to train the model  Traditional methods like Hidden Markov Model (HMM) with rule based approachesBIBREF9,BIBREF10, and Support Vector Machine (SVM) with manual feature-engineeringBIBREF11 have been applied but they perform poor compared to neural  However, there has been no research in Nepali NER using neural network  Therefore, we created the named entity annotated dataset partly with the help of Dataturk to train a neural model  The texts used for this dataset are collected from various daily news sources from Nepal around the year 2015-2016 
Following are our contributions:
We present a novel Named Entity Recognizer (NER) for Nepali language  To best of our knowledge we are the first to propose neural based Nepali NER 
As there are not good quality dataset to train NER we release a dataset to support future research
We perform empirical evaluation of our model with state-of-the-art models with relative improvement of upto 10%
In this paper, we present works similar to ours in Section SECREF2  We describe our approach and dataset statistics in Section SECREF3 and SECREF4, followed by our experiments, evaluation and discussion in Section SECREF5, SECREF6, and SECREF7  We conclude with our observations in Section SECREF8 
To facilitate further research our code and dataset will be made available at github com/link-yet-to-be-updated
Related Work
There has been a handful of research on Nepali NER task based on approaches like Support Vector Machine and gazetteer listBIBREF11 and Hidden Markov Model and gazetteer listBIBREF9,BIBREF10 
BIBREF11 uses SVM along with features like first word, word length, digit features and gazetteer (person, organization, location, middle name, verb, designation and others)  It uses one vs rest classification model to classify each word into different entity classes  However, it does not the take context word into account while training the model  Similarly, BIBREF9 and BIBREF10 uses Hidden Markov Model with n-gram technique for extracting POS-tags  POS-tags with common noun, proper noun or combination of both are combined together, then uses gazetteer list as look-up table to identify the named entities 
Researchers have shown that the neural networks like CNNBIBREF12, RNNBIBREF13, LSTMBIBREF14, GRUBIBREF15 can capture the semantic knowledge of language better with the help of pre-trained embbeddings like word2vecBIBREF16, gloveBIBREF17 or fasttextBIBREF18 
Similar approaches has been applied to many South Asian languages like HindiBIBREF6, IndonesianBIBREF7, BengaliBIBREF19 and In this paper, we present the neural network architecture for NER task in Nepali language, which doesn't require any manual feature engineering nor any data pre-processing during training  First we are comparing BiLSTMBIBREF14, BiLSTM+CNNBIBREF20, BiLSTM+CRFBIBREF1, BiLSTM+CNN+CRFBIBREF2 models with CNN modelBIBREF0 and Stanford CRF modelBIBREF21  Secondly, we show the comparison between models trained on general word embeddings, word embedding + character-level embedding, word embedding + part-of-speech(POS) one-hot encoding and word embedding + grapheme clustered or sub-word embeddingBIBREF22  The experiments were performed on the dataset that we created and on the dataset received from ILPRL lab  Our extensive study shows that augmenting word embedding with character or grapheme-level representation and POS one-hot encoding vector yields better results compared to using general word embedding alone 
Approach
In this section, we describe our approach in building our model  This model is partly inspired from multiple models BIBREF20,BIBREF1, andBIBREF2
Approach ::: Bidirectional LSTM
We used Bi-directional LSTM to capture the word representation in forward as well as reverse direction of a sentence  Generally, LSTMs take inputs from left (past) of the sentence and computes the hidden state  However, it is proven beneficialBIBREF23 to use bi-directional LSTM, where, hidden states are computed based from right (future) of sentence and both of these hidden states are concatenated to produce the final output as $h_t$=[$\overrightarrow{h_t}$;$\overleftarrow{h_t}$], where $\overrightarrow{h_t}$, $\overleftarrow{h_t}$ = hidden state computed in forward and backward direction respectively 
Approach ::: Features ::: Word embeddings
We have used Word2Vec BIBREF16, GloVe BIBREF17 and FastText BIBREF18 word vectors of 300 dimensions  These vectors were trained on the corpus obtained from Nepali National Corpus  This pre-lemmatized corpus consists of 14 million words from books, web-texts and news papers  This corpus was mixed with the texts from the dataset before training CBOW and skip-gram version of word2vec using gensim libraryBIBREF24  This trained model consists of vectors for 72782 unique words 
Light pre-processing was performed on the corpus before training it  For example, invalid characters or characters other than Devanagari were removed but punctuation and numbers were not removed  We set the window context at 10 and the rare words whose count is below 5 are dropped  These word embeddings were not frozen during the training session because fine-tuning word embedding help achieve better performance compared to frozen oneBIBREF20 
We have used fasttext embeddings in particular because of its sub-word representation ability, which is very useful in highly inflectional language as shown in Table TABREF25  We have trained the word embedding in such a way that the sub-word size remains between 1 and 4  We particularly chose this size because in Nepali language a single letter can also be a word, for example e, t, C, r, l, n, u and a single character (grapheme) or sub-word can be formed after mixture of dependent vowel signs with consonant letters for example, C + O + = CO, here three different consonant letters form a single sub-word 
The two-dimensional visualization of an example word npAl is shown in FIGREF14  Principal Component Analysis (PCA) technique was used to generate this visualization which helps use to analyze the nearest neighbor words of a given sample word  84 and 104 nearest neighbors were observed using word2vec and fasttext embedding respectively on the same corpus 
Approach ::: Features ::: Character-level embeddings
BIBREF20 and BIBREF2 successfully presented that the character-level embeddings, extracted using CNN, when combined with word embeddings enhances the NER model performance significantly, as it is able to capture morphological features of a word  Figure FIGREF7 shows the grapheme-level CNN used in our model, where inputs to CNN are graphemes  Character-level CNN is also built in similar fashion, except the inputs are characters  Grapheme or Character -level embeddings are randomly initialized from [0,1] with real values with uniform distribution of dimension 30 
Approach ::: Features ::: Grapheme-level embeddings
Grapheme is atomic meaningful unit in writing system of any languages  Since, Nepali language is highly morphologically inflectional, we compared grapheme-level representation with character-level representation to evaluate its effect  For example, in character-level embedding, each character of a word npAl results into n + + p + A + l has its own embedding  However, in grapheme level, a word npAl is clustered into graphemes, resulting into n + pA + l  Here, each grapheme has its own embedding  This grapheme-level embedding results good scores on par with character-level embedding in highly inflectional languages like Nepali, because graphemes also capture syntactic information similar to characters  We created grapheme clusters using uniseg package which is helpful in unicode text segmentations 
Approach ::: Features ::: Part-of-speech (POS) one hot encoding
We created one-hot encoded vector of POS tags and then concatenated with pre-trained word embeddings before passing it to BiLSTM network  A sample of data is shown in figure FIGREF13 
Dataset Statistics ::: OurNepali dataset
Since, we there was no publicly available standard Nepali NER dataset and did not receive any dataset from the previous researchers, we had to create our own dataset  This dataset contains the sentences collected from daily newspaper of the year 2015-2016  This dataset has three major classes Person (PER), Location (LOC) and Organization (ORG)  Pre-processing was performed on the text before creation of the dataset, for example all punctuations and numbers besides ',', '-', '|' and ' ' were removed  Currently, the dataset is in standard CoNLL-2003 IO formatBIBREF25 
Since, this dataset is not lemmatized originally, we lemmatized only the post-positions like Ek, kO, l, mA, m, my, jF, sg, aEG which are just the few examples among 299 post positions in Nepali language  We obtained these post-positions from sanjaalcorps and added few more to match our dataset  We will be releasing this list in our github repository  We found out that lemmatizing the post-positions boosted the F1 score by almost 10% 
In order to label our dataset with POS-tags, we first created POS annotated dataset of 6946 sentences and 16225 unique words extracted from POS-tagged Nepali National Corpus and trained a BiLSTM model with 95 14% accuracy which was used to create POS-tags for our dataset 
The dataset released in our github repository contains each word in newline with space separated POS-tags and Entity-tags  The sentences are separated by empty newline  A sample sentence from the dataset is presented in table FIGREF13 
Dataset Statistics ::: ILPRL dataset
After much time, we received the dataset from Bal Krishna Bal, ILPRL, KU  This dataset follows standard CoNLL-2003 IOB formatBIBREF25 with POS tags  This dataset is prepared by ILPRL Lab, KU and KEIV Technologies  Few corrections like correcting the NER tags had to be made on the dataset  The statistics of both the dataset is presented in table TABREF23 
Table TABREF24 presents the total entities (PER, LOC, ORG and MISC) from both of the dataset used in our experiments  The dataset is divided into three parts with 64%, 16% and 20% of the total dataset into training set, development set and test set respectively 
Experiments
In this section, we present the details about training our neural network  The neural network architecture are implemented using PyTorch framework BIBREF26  The training is performed on a single Nvidia Tesla P100 SXM2  We first run our experiment on BiLSTM, BiLSTM-CNN, BiLSTM-CRF BiLSTM-CNN-CRF using the hyper-parameters mentioned in Table TABREF30  The training and evaluation was done on sentence-level  The RNN variants are initialized randomly from $(-\sqrt{k},\sqrt{k})$ where $k=\frac{1}{hidden\_size}$ 
First we loaded our dataset and built vocabulary using torchtext library  This eased our process of data loading using its SequenceTaggingDataset class  We trained our model with shuffled training set using Adam optimizer with hyper-parameters mentioned in table TABREF30  All our models were trained on single layer of LSTM network  We found out Adam was giving better performance and faster convergence compared to Stochastic Gradient Descent (SGD)  We chose those hyper-parameters after many ablation studies  The dropout of 0 5 is applied after LSTM layer 
For CNN, we used 30 different filters of sizes 3, 4 and 5  The embeddings of each character or grapheme involved in a given word, were passed through the pipeline of Convolution, Rectified Linear Unit and Max-Pooling  The resulting vectors were concatenated and applied dropout of 0 5 before passing into linear layer to obtain the embedding size of 30 for the given word  This resulting embedding is concatenated with word embeddings, which is again concatenated with one-hot POS vector 
Experiments ::: Tagging Scheme
Currently, for our experiments we trained our model on IO (Inside, Outside) format for both the dataset, hence the dataset does not contain any B-type annotation unlike in BIO (Beginning, Inside, Outside) scheme 
Experiments ::: Early Stopping
We used simple early stopping technique where if the validation loss does not decrease after 10 epochs, the training was stopped, else the training will run upto 100 epochs  In our experience, training usually stops around 30-50 epochs 
Experiments ::: Hyper-parameters Tuning
We ran our experiment looking for the best hyper-parameters by changing learning rate from (0,1, 0 01, 0 001, 0 0001), weight decay from [$10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$, $10^{-7}$], batch size from [1, 2, 4, 8, 16, 32, 64, 128], hidden size from [8, 16, 32, 64, 128, 256, 512 1024]  Table TABREF30 shows all other hyper-parameter used in our experiment for both of the dataset 
Experiments ::: Effect of Dropout
Figure FIGREF31 shows how we end up choosing 0 5 as dropout rate  When the dropout layer was not used, the F1 score are at the lowest  As, we slowly increase the dropout rate, the F1 score also gradually increases, however after dropout rate = 0 5, the F1 score starts falling down  Therefore, we have chosen 0 5 as dropout rate for all other experiments performed 
Evaluation
In this section, we present the details regarding evaluation and comparison of our models with other baselines 
Table TABREF25 shows the study of various embeddings and comparison among each other in OurNepali dataset  Here, raw dataset represents such dataset where post-positions are not lemmatized  We can observe that pre-trained embeddings significantly improves the score compared to randomly initialized embedding  We can deduce that Skip Gram models perform better compared CBOW models for word2vec and fasttext  Here, fastText_Pretrained represents the embedding readily available in fastText website, while other embeddings are trained on the Nepali National Corpus as mentioned in sub-section SECREF11  From this table TABREF25, we can clearly observe that model using fastText_Skip Gram embeddings outperforms all other models 
Table TABREF35 shows the model architecture comparison between all the models experimented  The features used for Stanford CRF classifier are words, letter n-grams of upto length 6, previous word and next word  This model is trained till the current function value is less than $1\mathrm {e}{-2}$  The hyper-parameters of neural network experiments are set as shown in table TABREF30  Since, word embedding of character-level and grapheme-level is random, their scores are near 
All models are evaluated using CoNLL-2003 evaluation scriptBIBREF25 to calculate entity-wise precision, recall and f1 score 
Discussion
In this paper we present that we can exploit the power of neural network to train the model to perform downstream NLP tasks like Named Entity Recognition even in Nepali language  We showed that the word vectors learned through fasttext skip gram model performs better than other word embedding because of its capability to represent sub-word and this is particularly important to capture morphological structure of words and sentences in highly inflectional like Nepali  This concept can come handy in other Devanagari languages as well because the written scripts have similar syntactical structure 
We also found out that stemming post-positions can help a lot in improving model performance because of inflectional characteristics of Nepali language  So when we separate out its inflections or morphemes, we can minimize the variations of same word which gives its root word a stronger word vector representations compared to its inflected versions 
We can clearly imply from tables TABREF23, TABREF24, and TABREF35 that we need more data to get better results because OurNepali dataset volume is almost ten times bigger compared to ILPRL dataset in terms of entities 
Conclusion and Future work
In this paper, we proposed a novel NER for Nepali language and achieved relative improvement of upto 10% and studies different factors effecting the performance of the NER for Nepali language 
We also present a neural architecture BiLSTM+CNN(grapheme-level) which turns out to be performing on par with BiLSTM+CNN(character-level) under the same configuration  We believe this will not only help Nepali language but also other languages falling under the umbrellas of Devanagari languages  Our model BiLSTM+CNN(grapheme-level) and BiLSTM+CNN(G)+POS outperforms all other model experimented in OurNepali and ILPRL dataset respectively 
Since this is the first named entity recognition research in Nepal language using neural network, there are many rooms for improvement  We believe initializing the grapheme-level embedding with fasttext embeddings might help boosting the performance, rather than randomly initializing it  In future, we plan to apply other latest techniques like BERT, ELMo and FLAIR to study its effect on low-resource language like Nepali  We also plan to improve the model using cross-lingual or multi-lingual parameter sharing techniques by jointly training with other Devanagari languages like Hindi and Bengali 
Finally, we would like to contribute our dataset to Nepali NLP community to move forward the research going on in language understanding domain  We believe there should be special committee to create and maintain such dataset for Nepali NLP and organize various competitions which would elevate the NLP research in Nepal 
Some of the future works are listed below:
Proper initialization of grapheme level embedding from fasttext embeddings 
Apply robust POS-tagger for Nepali dataset
Lemmatize the OurNepali dataset with robust and efficient lemmatizer
Improve Nepali language score with cross-lingual learning techniques
Create more dataset using Wikipedia/Wikidata framework
Acknowledgments
The authors of this paper would like to express sincere thanks to Bal Krishna Bal, Kathmandu University Professor for providing us the POS-tagged Nepali NER data 
Context 3: Introduction
We understand from Zipf's Law that in any natural language corpus a majority of the vocabulary word types will either be absent or occur in low frequency  Estimating the statistical properties of these rare word types is naturally a difficult task  This is analogous to the curse of dimensionality when we deal with sequences of tokens - most sequences will occur only once in the training data  Neural network architectures overcome this problem by defining non-linear compositional models over vector space representations of tokens and hence assign non-zero probability even to sequences not seen during training BIBREF0 , BIBREF1   In this work, we explore a similar approach to learning distributed representations of social media posts by composing them from their constituent characters, with the goal of generalizing to out-of-vocabulary words as well as sequences at test time 
Traditional Neural Network Language Models (NNLMs) treat words as the basic units of language and assign independent vectors to each word type  To constrain memory requirements, the vocabulary size is fixed before-hand; therefore, rare and out-of-vocabulary words are all grouped together under a common type `UNKNOWN'  This choice is motivated by the assumption of arbitrariness in language, which means that surface forms of words have little to do with their semantic roles  Recently, BIBREF2 challenge this assumption and present a bidirectional Long Short Term Memory (LSTM) BIBREF3 for composing word vectors from their constituent characters which can memorize the arbitrary aspects of word orthography as well as generalize to rare and out-of-vocabulary words 
Encouraged by their findings, we extend their approach to a much larger unicode character set, and model long sequences of text as functions of their constituent characters (including white-space)  We focus on social media posts from the website Twitter, which are an excellent testing ground for character based models due to the noisy nature of text  Heavy use of slang and abundant misspellings means that there are many orthographically and semantically similar tokens, and special characters such as emojis are also immensely popular and carry useful semantic information  In our moderately sized training dataset of 2 million tweets, there were about 0 92 million unique word types  It would be expensive to capture all these phenomena in a word based model in terms of both the memory requirement (for the increased vocabulary) and the amount of training data required for effective learning  Additional benefits of the character based approach include language independence of the methods, and no requirement of NLP preprocessing such as word-segmentation 
A crucial step in learning good text representations is to choose an appropriate objective function to optimize  Unsupervised approaches attempt to reconstruct the original text from its latent representation BIBREF4 , BIBREF0   Social media posts however, come with their own form of supervision annotated by millions of users, in the form of hashtags which link posts about the same topic together  A natural assumption is that the posts with the same hashtags should have embeddings which are close to each other  Hence, we formulate our training objective to maximize cross-entropy loss at the task of predicting hashtags for a post from its latent representation 
We propose a Bi-directional Gated Recurrent Unit (Bi-GRU) BIBREF5 neural network for learning tweet representations  Treating white-space as a special character itself, the model does a forward and backward pass over the entire sequence, and the final GRU states are linearly combined to get the tweet embedding  Posterior probabilities over hashtags are computed by projecting this embedding to a softmax output layer  Compared to a word-level baseline this model shows improved performance at predicting hashtags for a held-out set of posts  Inspired by recent work in learning vector space text representations, we name our model tweet2vec 
Related Work
Using neural networks to learn distributed representations of words dates back to BIBREF0   More recently, BIBREF4 released word2vec - a collection of word vectors trained using a recurrent neural network  These word vectors are in widespread use in the NLP community, and the original work has since been extended to sentences BIBREF1 , documents and paragraphs BIBREF6 , topics BIBREF7 and queries BIBREF8   All these methods require storing an extremely large table of vectors for all word types and cannot be easily generalized to unseen words at test time BIBREF2   They also require preprocessing to find word boundaries which is non-trivial for a social network domain like Twitter 
In BIBREF2 , the authors present a compositional character model based on bidirectional LSTMs as a potential solution to these problems  A major benefit of this approach is that large word lookup tables can be compacted into character lookup tables and the compositional model scales to large data sets better than other state-of-the-art approaches  While BIBREF2 generate word embeddings from character representations, we propose to generate vector representations of entire tweets from characters in our tweet2vec model 
Our work adds to the growing body of work showing the applicability of character models for a variety of NLP tasks such as Named Entity Recognition BIBREF9 , POS tagging BIBREF10 , text classification BIBREF11 and language modeling BIBREF12 , BIBREF13  
Previously, BIBREF14 dealt with the problem of estimating rare word representations by building them from their constituent morphemes  While they show improved performance over word-based models, their approach requires a morpheme parser for preprocessing which may not perform well on noisy text like Twitter  Also the space of all morphemes, though smaller than the space of all words, is still large enough that modelling all morphemes is impractical 
Hashtag prediction for social media has been addressed earlier, for example in BIBREF15 , BIBREF16   BIBREF15 also use a neural architecture, but compose text embeddings from a lookup table of words  They also show that the learned embeddings can generalize to an unrelated task of document recommendation, justifying the use of hashtags as supervision for learning text representations 
Tweet2Vec
Bi-GRU Encoder: Figure 1 shows our model for encoding tweets  It uses a similar structure to the C2W model in BIBREF2 , with LSTM units replaced with GRU units

ground_truth: OurNepali contains 3 different types of entities, ILPRL contains 4 different types of entities

three

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What limitations do the authors demnostrate of their model?
Context 2: Introduction
Neural networks have been successfully used to describe images with text using sequence-to-sequence models BIBREF0  However, the results are simple and dry captions which are one or two phrases long  Humans looking at a painting see more than just objects  Paintings stimulate sentiments, metaphors and stories as well  Therefore, our goal is to have a neural network describe the painting artistically in a style of choice  As a proof of concept, we present a model which generates Shakespearean prose for a given painting as shown in Figure FIGREF1  Accomplishing this task is difficult with traditional sequence to sequence models since there does not exist a large collection of Shakespearean prose which describes paintings: Shakespeare's works describes a single painting shown in Figure FIGREF3  Fortunately we have a dataset of modern English poems which describe images BIBREF1 and line-by-line modern paraphrases of Shakespeare's plays BIBREF2  Our solution is therefore to combine two separately trained models to synthesize Shakespearean prose for a given painting 
Introduction ::: Related work
A general end-to-end approach to sequence learning BIBREF3 makes minimal assumptions on the sequence structure  This model is widely used in tasks such as machine translation, text summarization, conversational modeling, and image captioning  A generative model using a deep recurrent architecture BIBREF0 has also beeen used for generating phrases describing an image  The task of synthesizing multiple lines of poetry for a given image BIBREF1 is accomplished by extracting poetic clues from images  Given the context image, the network associates image attributes with poetic descriptions using a convolutional neural net  The poem is generated using a recurrent neural net which is trained using multi-adversarial training via policy gradient 
Transforming text from modern English to Shakespearean English using text "style transfer" is challenging  An end to end approach using a sequence-to-sequence model over a parallel text corpus BIBREF2 has been proposed based on machine translation  In the absence of a parallel text corpus, generative adversarial networks (GANs) have been used, which simultaneously train two models: a generative model which captures the data distribution, and a discriminative model which evaluates the performance of the generator  Using a target domain language model as a discriminator has also been employed BIBREF4, providing richer and more stable token-level feedback during the learning process  A key challenge in both image and text style transfer is separating content from style BIBREF5, BIBREF6, BIBREF7  Cross-aligned auto-encoder models have focused on style transfer using non-parallel text BIBREF7  Recently, a fine grained model for text style transfer has been proposed BIBREF8 which controls several factors of variation in textual data by using back-translation  This allows control over multiple attributes, such as gender and sentiment, and fine-grained control over the trade-off between content and style 
Methods
We use a total three datasets: two datasets for generating an English poem from an image, and Shakespeare plays and their English translations for text style transfer 
We train a model for generating poems from images based on two datasets BIBREF1  The first dataset consists of image and poem pairs, namely a multi-modal poem dataset (MultiM-Poem), and the second dataset is a large poem corpus, namely a uni-modal poem dataset (UniM-Poem)  The image and poem pairs are extended by adding the nearest three neighbor poems from the poem corpus without redundancy, and an extended image and poem pair dataset is constructed and denoted as MultiM-Poem(Ex)BIBREF1 
We use a collection of line-by-line modern paraphrases for 16 of Shakespeare’s plays BIBREF2, for training a style transfer network from English poems to Shakespearean prose  We use 18,395 sentences from the training data split  We keep 1,218 sentences in the validation data set and 1,462 sentences in our test set 
Methods ::: Image To Poem Actor-Critic Model
For generating a poem from images we use an existing actor-critic architecture BIBREF1  This involves 3 parallel CNNs: an object CNN, sentiment CNN, and scene CNN, for feature extraction  These features are combined with a skip-thought model which provides poetic clues, which are then fed into a sequence-to-sequence model trained by policy gradient with 2 discriminator networks for rewards  This as a whole forms a pipeline that takes in an image and outputs a poem as shown on the top left of Figure FIGREF4  A CNN-RNN generative model acts as an agent  The parameters of this agent define a policy whose execution determines which word is selected as an action  When the agent selects all words in a poem, it receives a reward  Two discriminative networks, shown on the top right of Figure FIGREF4, are defined to serve as rewards concerning whether the generated poem properly describes the input image and whether the generated poem is poetic  The goal of the poem generation model is to generate a sequence of words as a poem for an image to maximize the expected return 
Methods ::: Shakespearizing Poetic Captions
For Shakespearizing modern English texts, we experimented with various types of sequence to sequence models  Since the size of the parallel translation data available is small, we leverage a dictionary providing a mapping between Shakespearean words and modern English words to retrofit pre-trained word embeddings  Incorporating this extra information improves the translation task  The large number of shared word types between the source and target sentences indicates that sharing the representation between them is beneficial 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with Attention
We use a sequence-to-sequence model which consists of a single layer unidrectional LSTM encoder and a single layer LSTM decoder and pre-trained retrofitted word embeddings shared between source and target sentences  We experimented with two different types of attention: global attention BIBREF9, in which the model makes use of the output from the encoder and decoder for the current time step only, and Bahdanau attention BIBREF10, where computing attention requires the output of the decoder from the prior time step  We found that global attention performs better in practice for our task of text style transfer 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with a Pointer Network
Since a pair of corresponding Shakespeare and modern English sentences have significant vocabulary overlap we extend the sequence-to-sequence model mentioned above using pointer networks BIBREF11 that provide location based attention and have been used to enable copying of tokens directly from the input  Moreover, there are lot of proper nouns and rare words which might not be predicted by a vanilla sequence to sequence model 
Methods ::: Shakespearizing Poetic Captions ::: Prediction
For both seq2seq models, we use the attention matrices returned at each decoder time step during inference, to compute the next word in the translated sequence if the decoder output at the current time step is the UNK token  We replace the UNKs in the target output with the highest aligned, maximum attention, source word  The seq2seq model with global attention gives the best results with an average target BLEU score of 29 65 on the style transfer dataset, compared with an average target BLEU score of 26 97 using the seq2seq model with pointer networks 
Results
We perform a qualitative analysis of the Shakespearean prose generated for the input paintings  We conducted a survey, in which we presented famous paintings including those shown in Figures FIGREF1 and FIGREF10 and the corresponding Shakespearean prose generated by the model, and asked 32 students to rate them on the basis of content, creativity and similarity to Shakespearean style on a Likert scale of 1-5  Figure FIGREF12 shows the result of our human evaluation 
The average content score across the paintings is 3 7 which demonstrates that the prose generated is relevant to the painting  The average creativity score is 3 9 which demonstrates that the model captures more than basic objects in the painting successfully using poetic clues in the scene  The average style score is 3 9 which demonstrates that the prose generated is perceived to be in the style of Shakespeare 
We also perform a quantitative analysis of style transfer by generating BLEU scores for the model output using the style transfer dataset  The variation of the BLEU scores with the source sentence lengths is shown in Figure FIGREF11  As expected, the BLEU scores decrease with increase in source sentence lengths 
Results ::: Implementation
All models were trained on Google Colab with a single GPU using Python 3 6 and Tensorflow 2 0  The number of hidden units for the encoder and decoder is 1,576 and 256 for seq2seq with global attention and seq2seq with pointer networks respectively  Adam optimizer was used with the default learning rate of 0 001  The model was trained for 25 epochs  We use pre-trained retrofitted word embeddings of dimension 192 
Results ::: Limitations
Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for "Starry Night" with a low average content score  This happens when the style transfer dataset does not have similar words in the training set of sentences  A solution would be to expand the style transfer dataset, for a better representation of the poem data 
Results ::: Conclusions and Future Work
In conclusion, combining two pipelines with an intermediate representation works well in practice  We observe that a CNN-RNN based image-to-poem net combined with a seq2seq model with parallel text corpus for text style transfer synthesizes Shakespeare-style prose for a given painting  For the seq2seq model used, we observe that it performs better in practice using global attention as compared with local attention  We make our models and code publicly available BIBREF12  In future work we would like to experiment with GANs in the absence of non-parallel datasets, so that we can use varied styles for text style transfer  We would also like to experiment with cross aligned auto-encoders, which form a latent content representation, to efficiently separate style and content 
Context 3: 
The input to the network is defined by an alphabet of characters $C$ (this may include the entire unicode character set)  The input tweet is broken into a stream of characters $c_1, c_2,     c_m$ each of which is represented by a 1-by- $|C|$ encoding  These one-hot vectors are then projected to a character space by multiplying with the matrix $P_C \in \mathbb {R}^{|C| \times d_c}$ , where $d_c$ is the dimension of the character vector space  Let $x_1, x_2,     x_m$ be the sequence of character vectors for the input tweet after the lookup  The encoder consists of a forward-GRU and a backward-GRU  Both have the same architecture, except the backward-GRU processes the sequence in reverse order  Each of the GRU units process these vectors sequentially, and starting with the initial state $h_0$ compute the sequence $h_1, h_2,     h_m$ as follows: $ r_t &= \sigma (W_r x_t + U_r h_{t-1} + b_r), \\ z_t &= \sigma (W_z x_t + U_z h_{t-1} + b_z), \\ \tilde{h}_t &= tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h), \\ h_t &= (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t  $
Here $r_t$ , $z_t$ are called the reset and update gates respectively, and $\tilde{h}_t$ is the candidate output state which is converted to the actual output state $h_t$   $W_r, W_z, W_h$ are $d_h \times d_c$ matrices and $U_r, U_z, U_h$ are $d_h \times d_h$ matrices, where $d_h$ is the hidden state dimension of the GRU  The final states $h_m^f$ from the forward-GRU, and $z_t$0 from the backward GRU are combined using a fully-connected layer to the give the final tweet embedding $z_t$1 :
$$e_t = W^f h_m^f + W^b h_0^b$$   (Eq  3)
Here $W^f, W^b$ are $d_t \times d_h$ and $b$ is $d_t \times 1$ bias term, where $d_t$ is the dimension of the final tweet embedding  In our experiments we set $d_t=d_h$   All parameters are learned using gradient descent 
Softmax: Finally, the tweet embedding is passed through a linear layer whose output is the same size as the number of hashtags $L$ in the data set  We use a softmax layer to compute the posterior hashtag probabilities:
$$P(y=j |e) = \frac{exp(w_j^Te + b_j)}{\sum _{i=1}^L exp(w_i^Te + b_j)} $$   (Eq  4)
Objective Function: We optimize the categorical cross-entropy loss between predicted and true hashtags:
$$J = \frac{1}{B} \sum _{i=1}^{B} \sum _{j=1}^{L} -t_{i,j}log(p_{i,j}) + \lambda \Vert \Theta \Vert ^2 $$   (Eq  5)
Here $B$ is the batch size, $L$ is the number of classes, $p_{i,j}$ is the predicted probability that the $i$ -th tweet has hashtag $j$ , and $t_{i,j} \in \lbrace 0,1\rbrace $ denotes the ground truth of whether the $j$ -th hashtag is in the $i$ -th tweet  We use L2-regularization weighted by $\lambda $  
Word Level Baseline
Since our objective is to compare character-based and word-based approaches, we have also implemented a simple word-level encoder for tweets  The input tweet is first split into tokens along white-spaces  A more sophisticated tokenizer may be used, but for a fair comparison we wanted to keep language specific preprocessing to a minimum  The encoder is essentially the same as tweet2vec, with the input as words instead of characters  A lookup table stores word vectors for the $V$ (20K here) most common words, and the rest are grouped together under the `UNK' token 
Data
Our dataset consists of a large collection of global posts from Twitter between the dates of June 1, 2013 to June 5, 2013  Only English language posts (as detected by the lang field in Twitter API) and posts with at least one hashtag are retained  We removed infrequent hashtags ( $<500$ posts) since they do not have enough data for good generalization  We also removed very frequent tags ( $>19K$ posts) which were almost always from automatically generated posts (ex: #androidgame) which are trivial to predict  The final dataset contains 2 million tweets for training, 10K for validation and 50K for testing, with a total of 2039 distinct hashtags  We use simple regex to preprocess the post text and remove hashtags (since these are to be predicted) and HTML tags, and replace user-names and URLs with special tokens  We also removed retweets and convert the text to lower-case 
Implementation Details
Word vectors and character vectors are both set to size $d_L=150$ for their respective models  There were 2829 unique characters in the training set and we model each of these independently in a character look-up table

ground_truth: Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer

we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for "Starry Night" with a low average content score

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What other tasks do they test their method on?
Context 3:  Embedding sizes were chosen such that each model had roughly the same number of parameters (Table 2 )  Training is performed using mini-batch gradient descent with Nesterov's momentum  We use a batch size $B=64$ , initial learning rate $\eta _0=0 01$ and momentum parameter $\mu _0=0 9$   L2-regularization with $\lambda =0 001$ was applied to all models  Initial weights were drawn from 0-mean gaussians with $\sigma =0 1$ and initial biases were set to 0  The hyperparameters were tuned one at a time keeping others fixed, and values with the lowest validation cost were chosen  The resultant combination was used to train the models until performance on validation set stopped increasing  During training, the learning rate is halved everytime the validation set precision increases by less than 0 01 % from one epoch to the next  The models converge in about 20 epochs  Code for training both the models is publicly available on github 
Results
We test the character and word-level variants by predicting hashtags for a held-out test set of posts  Since there may be more than one correct hashtag per post, we generate a ranked list of tags for each post from the output posteriors, and report average precision@1, recall@10 and mean rank of the correct hashtags  These are listed in Table 3  
To see the performance of each model on posts containing rare words (RW) and frequent words (FW) we selected two test sets each containing 2,000 posts  We populated these sets with posts which had the maximum and minimum number of out-of-vocabulary words respectively, where vocabulary is defined by the 20K most frequent words  Overall, tweet2vec outperforms the word model, doing significantly better on RW test set and comparably on FW set  This improved performance comes at the cost of increased training time (see Table 2 ), since moving from words to characters results in longer input sequences to the GRU 
We also study the effect of model size on the performance of these models  For the word model we set vocabulary size $V$ to 8K, 15K and 20K respectively  For tweet2vec we set the GRU hidden state size to 300, 400 and 500 respectively  Figure 2 shows precision 1 of the two models as the number of parameters is increased, for each test set described above  There is not much variation in the performance, and moreover tweet2vec always outperforms the word based model for the same number of parameters 
Table 4 compares the models as complexity of the task is increased  We created 3 datasets (small, medium and large) with an increasing number of hashtags to be predicted  This was done by varying the lower threshold of the minimum number of tags per post for it to be included in the dataset  Once again we observe that tweet2vec outperforms its word-based counterpart for each of the three settings 
Finally, table 1 shows some predictions from the word level model and tweet2vec  We selected these to highlight some strengths of the character based approach - it is robust to word segmentation errors and spelling mistakes, effectively interprets emojis and other special characters to make predictions, and also performs comparably to the word-based approach for in-vocabulary tokens 
Conclusion
We have presented tweet2vec - a character level encoder for social media posts trained using supervision from associated hashtags  Our result shows that tweet2vec outperforms the word based approach, doing significantly better when the input post contains many rare words  We have focused only on English language posts, but the character model requires no language specific preprocessing and can be extended to other languages  For future work, one natural extension would be to use a character-level decoder for predicting the hashtags  This will allow generation of hashtags not seen in the training dataset  Also, it will be interesting to see how our tweet2vec embeddings can be used in domains where there is a need for semantic understanding of social media, such as tracking infectious diseases BIBREF17   Hence, we provide an off-the-shelf encoder trained on medium dataset described above to compute vector-space representations of tweets along with our code on github 
Acknowledgments
We would like to thank Alex Smola, Yun Fu, Hsiao-Yu Fish Tung, Ruslan Salakhutdinov, and Barnabas Poczos for useful discussions  We would also like to thank Juergen Pfeffer for providing access to the Twitter data, and the reviewers for their comments 

ground_truth: None