prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How is the dataset of hashtags sourced?
Context 1: Introduction
A hashtag is a keyphrase represented as a sequence of alphanumeric characters plus underscore, preceded by the # symbol  Hashtags play a central role in online communication by providing a tool to categorize the millions of posts generated daily on Twitter, Instagram, etc  They are useful in search, tracking content about a certain topic BIBREF0 , BIBREF1 , or discovering emerging trends BIBREF2  
Hashtags often carry very important information, such as emotion BIBREF3 , sentiment BIBREF4 , sarcasm BIBREF5 , and named entities BIBREF6 , BIBREF7   However, inferring the semantics of hashtags is non-trivial since many hashtags contain multiple tokens joined together, which frequently leads to multiple potential interpretations (e g , lion head vs  lionhead)  Table TABREF3 shows several examples of single- and multi-token hashtags  While most hashtags represent a mix of standard tokens, named entities and event names are prevalent and pose challenges to both human and automatic comprehension, as these are more likely to be rare tokens  Hashtags also tend to be shorter to allow fast typing, to attract attention or to satisfy length limitations imposed by some social media platforms  Thus, they tend to contain a large number of abbreviations or non-standard spelling variations (e g , #iloveu4eva) BIBREF8 , BIBREF9 , which hinders their understanding 
The goal of our study is to build efficient methods for automatically splitting a hashtag into a meaningful word sequence  Our contributions are:
Our new dataset includes segmentation for 12,594 unique hashtags and their associated tweets annotated in a multi-step process for higher quality than the previous dataset of 1,108 hashtags BIBREF10   We frame the segmentation task as a pairwise ranking problem, given a set of candidate segmentations  We build several neural architectures using this problem formulation which use corpus-based, linguistic and thesaurus based features  We further propose a multi-task learning approach which jointly learns segment ranking and single- vs  multi-token hashtag classification  The latter leads to an error reduction of 24 6% over the current state-of-the-art  Finally, we demonstrate the utility of our method by using hashtag segmentation in the downstream task of sentiment analysis  Feeding the automatically segmented hashtags to a state-of-the-art sentiment analysis method on the SemEval 2017 benchmark dataset results in a 2 6% increase in the official metric for the task 
Background and Preliminaries
Current approaches for hashtag segmentation can be broadly divided into three categories: (a) gazeteer and rule based BIBREF11 , BIBREF12 , BIBREF13 , (b) word boundary detection BIBREF14 , BIBREF15 , and (c) ranking with language model and other features BIBREF16 , BIBREF10 , BIBREF0 , BIBREF17 , BIBREF18   Hashtag segmentation approaches draw upon work on compound splitting for languages such as German or Finnish BIBREF19 and word segmentation BIBREF20 for languages with no spaces between words such as Chinese BIBREF21 , BIBREF22   Similar to our work, BIBREF10 BansalBV15 extract an initial set of candidate segmentations using a sliding window, then rerank them using a linear regression model trained on lexical, bigram and other corpus-based features  The current state-of-the-art approach BIBREF14 , BIBREF15 uses maximum entropy and CRF models with a combination of language model and hand-crafted features to predict if each character in the hashtag is the beginning of a new word 
Generating Candidate Segmentations  Microsoft Word Breaker BIBREF16 is, among the existing methods, a strong baseline for hashtag segmentation, as reported in BIBREF14 and BIBREF10   It employs a beam search algorithm to extract INLINEFORM0 best segmentations as ranked by the n-gram language model probability: INLINEFORM1
where INLINEFORM0 is the word sequence of segmentation INLINEFORM1 and INLINEFORM2 is the window size  More sophisticated ranking strategies, such as Binomial and word length distribution based ranking, did not lead to a further improvement in performance BIBREF16   The original Word Breaker was designed for segmenting URLs using language models trained on web data  In this paper, we reimplemented and tailored this approach to segmenting hashtags by using a language model specifically trained on Twitter data (implementation details in § SECREF26 )  The performance of this method itself is competitive with state-of-the-art methods (evaluation results in § SECREF46 )  Our proposed pairwise ranking method will effectively take the top INLINEFORM3 segmentations generated by this baseline as candidates for reranking 
However, in prior work, the ranking scores of each segmentation were calculated independently, ignoring the relative order among the top INLINEFORM0 candidate segmentations  To address this limitation, we utilize a pairwise ranking strategy for the first time for this task and propose neural architectures to model this 
Multi-task Pairwise Neural Ranking
We propose a multi-task pairwise neural ranking approach to better incorporate and distinguish the relative order between the candidate segmentations of a given hashtag  Our model adapts to address single- and multi-token hashtags differently via a multi-task learning strategy without requiring additional annotations  In this section, we describe the task setup and three variants of pairwise neural ranking models (Figure FIGREF11 ) 
Segmentation as Pairwise Ranking
The goal of hashtag segmentation is to divide a given hashtag INLINEFORM0 into a sequence of meaningful words INLINEFORM1   For a hashtag of INLINEFORM2 characters, there are a total of INLINEFORM3 possible segmentations but only one, or occasionally two, of them ( INLINEFORM4 ) are considered correct (Table TABREF9 ) 
We transform this task into a pairwise ranking problem: given INLINEFORM0 candidate segmentations { INLINEFORM1 }, we rank them by comparing each with the rest in a pairwise manner  More specifically, we train a model to predict a real number INLINEFORM2 for any two candidate segmentations INLINEFORM3 and INLINEFORM4 of hashtag INLINEFORM5 , which indicates INLINEFORM6 is a better segmentation than INLINEFORM7 if positive, and vice versa  To quantify the quality of a segmentation in training, we define a gold scoring function INLINEFORM8 based on the similarities with the ground-truth segmentation INLINEFORM9 : INLINEFORM10
We use the Levenshtein distance (minimum number of single-character edits) in this paper, although it is possible to use other similarity measurements as alternatives  We use the top INLINEFORM0 segmentations generated by Microsoft Word Breaker (§ SECREF2 ) as initial candidates 
Pairwise Neural Ranking Model
For an input candidate segmentation pair INLINEFORM0 , we concatenate their feature vectors INLINEFORM1 and INLINEFORM2 , and feed them into a feedforward network which emits a comparison score INLINEFORM3   The feature vector INLINEFORM4 or INLINEFORM5 consists of language model probabilities using Good-Turing BIBREF23 and modified Kneser-Ney smoothing BIBREF24 , BIBREF25 , lexical and linguistic features (more details in § SECREF23 )  For training, we use all the possible pairs INLINEFORM6 of the INLINEFORM7 candidates as the input and their gold scores INLINEFORM8 as the target  The training objective is to minimize the Mean Squared Error (MSE): DISPLAYFORM0
where INLINEFORM0 is the number of training examples 
To aggregate the pairwise comparisons, we follow a greedy algorithm proposed by BIBREF26 cohen1998learning and used for preference ranking BIBREF27   For each segmentation INLINEFORM0 in the candidate set INLINEFORM1 , we calculate a single score INLINEFORM2 , and find the segmentation INLINEFORM3 corresponding to the highest score  We repeat the same procedure after removing INLINEFORM4 from INLINEFORM5 , and continue until INLINEFORM6 reduces to an empty set  Figure FIGREF11 (a) shows the architecture of this model 
Margin Ranking (MR) Loss
As an alternative to the pairwise ranker (§ SECREF15 ), we propose a pairwise model which learns from candidate pairs INLINEFORM0 but ranks each individual candidate directly rather than relatively  We define a new scoring function INLINEFORM1 which assigns a higher score to the better candidate, i e , INLINEFORM2 , if INLINEFORM3 is a better candidate than INLINEFORM4 and vice-versa  Instead of concatenating the features vectors INLINEFORM5 and INLINEFORM6 , we feed them separately into two identical feedforward networks with shared parameters  During testing, we use only one of the networks to rank the candidates based on the INLINEFORM7 scores  For training, we add a ranking layer on top of the networks to measure the violations in the ranking order and minimize the Margin Ranking Loss (MR): DISPLAYFORM0
where INLINEFORM0 is the number of training samples  The architecture of this model is presented in Figure FIGREF11 (b) 
Adaptive Multi-task Learning
Both models in § SECREF15 and § SECREF17 treat all the hashtags uniformly  However, different features address different types of hashtags  By design, the linguistic features capture named entities and multi-word hashtags that exhibit word shape patterns, such as camel case  The ngram probabilities with Good-Turing smoothing gravitate towards multi-word segmentations with known words, as its estimate for unseen ngrams depends on the fraction of ngrams seen once which can be very low BIBREF28   The modified Kneser-Ney smoothing is more likely to favor segmentations that contain rare words, and single-word segmentations in particular  Please refer to § SECREF46 for a more detailed quantitative and qualitative analysis 
To leverage this intuition, we introduce a binary classification task to help the model differentiate single-word from multi-word hashtags  The binary classifier takes hashtag features INLINEFORM0 as the input and outputs INLINEFORM1 , which represents the probability of INLINEFORM2 being a multi-word hashtag  INLINEFORM3 is used as an adaptive gating value in our multi-task learning setup  The gold labels for this task are obtained at no extra cost by simply verifying whether the ground-truth segmentation has multiple words  We train the pairwise segmentation ranker and the binary single- vs  multi-token hashtag classifier jointly, by minimizing INLINEFORM4 for the pairwise ranker and the Binary Cross Entropy Error ( INLINEFORM5 ) for the classifier: DISPLAYFORM0
where INLINEFORM0 is the adaptive gating value, INLINEFORM1 indicates if INLINEFORM2 is actually a multi-word hashtag and INLINEFORM3 is the number of training examples  INLINEFORM4 and INLINEFORM5 are the weights for each loss  For our experiments, we apply equal weights 
More specifically, we divide the segmentation feature vector INLINEFORM0 into two subsets: (a) INLINEFORM1 with modified Kneser-Ney smoothing features, and (b) INLINEFORM2 with Good-Turing smoothing and linguistic features  For an input candidate segmentation pair INLINEFORM3 , we construct two pairwise vectors INLINEFORM4 and INLINEFORM5 by concatenation, then combine them based on the adaptive gating value INLINEFORM6 before feeding them into the feedforward network INLINEFORM7 for pairwise ranking: DISPLAYFORM0
We use summation with padding, as we find this simple ensemble method achieves similar performance in our experiments as the more complex multi-column networks BIBREF29   Figure FIGREF11 (c) shows the architecture of this model  An analogue multi-task formulation can also be used for the Margin Ranking loss as: DISPLAYFORM0
Features
We use a combination of corpus-based and linguistic features to rank the segmentations  For a candidate segmentation INLINEFORM0 , its feature vector INLINEFORM1 includes the number of words in the candidate, the length of each word, the proportion of words in an English dictionary or Urban Dictionary BIBREF30 , ngram counts from Google Web 1TB corpus BIBREF31 , and ngram probabilities from trigram language models trained on the Gigaword corpus BIBREF32 and 1 1 billion English tweets from 2010, respectively  We train two language models on each corpus: one with Good-Turing smoothing using SRILM BIBREF33 and the other with modified Kneser-Ney smoothing using KenLM BIBREF34   We also add boolean features, such as if the candidate is a named-entity present in the list of Wikipedia titles, and if the candidate segmentation INLINEFORM2 and its corresponding hashtag INLINEFORM3 satisfy certain word-shapes (more details in appendix SECREF61 ) 
Similarly, for hashtag INLINEFORM0 , we extract the feature vector INLINEFORM1 consisting of hashtag length, ngram count of the hashtag in Google 1TB corpus BIBREF31 , and boolean features indicating if the hashtag is in an English dictionary or Urban Dictionary, is a named-entity, is in camel case, ends with a number, and has all the letters as consonants  We also include features of the best-ranked candidate by the Word Breaker model 
Implementation Details
We use the PyTorch framework to implement our multi-task pairwise ranking model  The pairwise ranker consists of an input layer, three hidden layers with eight nodes in each layer and hyperbolic tangent ( INLINEFORM0 ) activation, and a single linear output node  The auxiliary classifier consists of an input layer, one hidden layer with eight nodes and one output node with sigmoid activation  We use the Adam algorithm BIBREF35 for optimization and apply a dropout of 0 5 to prevent overfitting  We set the learning rate to 0 01 and 0 05 for the pairwise ranker and auxiliary classifier respectively  For each experiment, we report results obtained after 100 epochs 
For the baseline model used to extract the INLINEFORM0 initial candidates, we reimplementated the Word Breaker BIBREF16 as described in § SECREF2 and adapted it to use a language model trained on 1 1 billion tweets with Good-Turing smoothing using SRILM BIBREF33 to give a better performance in segmenting hashtags (§ SECREF46 )  For all our experiments, we set INLINEFORM1  
Hashtag Segmentation Data
We use two datasets for experiments (Table TABREF29 ): (a) STAN INLINEFORM0 , created by BIBREF10 BansalBV15, which consists of 1,108 unique English hashtags from 1,268 randomly selected tweets in the Stanford Sentiment Analysis Dataset BIBREF36 along with their crowdsourced segmentations and our additional corrections; and (b) STAN INLINEFORM1 , our new expert curated dataset, which includes all 12,594 unique English hashtags and their associated tweets from the same Stanford dataset 
Experiments
In this section, we present experimental results that compare our proposed method with the other state-of-the-art approaches on hashtag segmentation datasets  The next section will show experiments of applying hashtag segmentation to the popular task of sentiment analysis 
Existing Methods
We compare our pairwise neural ranker with the following baseline and state-of-the-art approaches:
The original hashtag as a single token;
A rule-based segmenter, which employs a set of word-shape rules with an English dictionary BIBREF13 ;
A Viterbi model which uses word frequencies from a book corpus BIBREF0 ;
The specially developed GATE Hashtag Tokenizer from the open source toolkit, which combines dictionaries and gazetteers in a Viterbi-like algorithm BIBREF11 ;
A maximum entropy classifier (MaxEnt) trained on the STAN INLINEFORM0 training dataset  It predicts whether a space should be inserted at each position in the hashtag and is the current state-of-the-art BIBREF14 ;
Our reimplementation of the Word Breaker algorithm which uses beam search and a Twitter ngram language model BIBREF16 ;
A pairwise linear ranker which we implemented for comparison purposes with the same features as our neural model, but using perceptron as the underlying classifier BIBREF38 and minimizing the hinge loss between INLINEFORM0 and a scoring function similar to INLINEFORM1   It is trained on the STAN INLINEFORM2 dataset 
Evaluation Metrics
We evaluate the performance by the top INLINEFORM0 ( INLINEFORM1 ) accuracy (A@1, A@2), average token-level F INLINEFORM2 score (F INLINEFORM3 @1), and mean reciprocal rank (MRR)  In particular, the accuracy and MRR are calculated at the segmentation-level, which means that an output segmentation is considered correct if and only if it fully matches the human segmentation  Average token-level F INLINEFORM4 score accounts for partially correct segmentation in the multi-token hashtag cases 
Results
Tables TABREF32 and TABREF33 show the results on the STAN INLINEFORM0 and STAN INLINEFORM1 datasets, respectively  All of our pairwise neural rankers are trained on the 2,518 manually segmented hashtags in the training set of STAN INLINEFORM2 and perform favorably against other state-of-the-art approaches  Our best model (MSE+multitask) that utilizes different features adaptively via a multi-task learning procedure is shown to perform better than simply combining all the features together (MR and MSE)  We highlight the 24 6% error reduction on STAN INLINEFORM3 and 16 5% on STAN INLINEFORM4 of our approach over the previous SOTA BIBREF14 on the Multi-token hashtags, and the importance of having a separate evaluation of multi-word cases as it is trivial to obtain 100% accuracy for Single-token hashtags  While our hashtag segmentation model is achieving a very high accuracy@2, to be practically useful, it remains a challenge to get the top one predication exactly correct  Some hashtags are very difficult to interpret, e g , #BTVbrownSMB refers to the Social Media Breakfast (SMB) in Burlington, Vermont (BTV) 
The improved Word Breaker with our addition of a Twitter-specific language model is a very strong baseline, which echos the findings of the original Word Breaker paper BIBREF16 that having a large in-domain language model is extremely helpful for word segmentation tasks  It is worth noting that the other state-of-the-art system BIBREF14 also utilized a 4-gram language model trained on 476 million tweets from 2009 
Analysis and Discussion
To empirically illustrate the effectiveness of different features on different types of hashtags, we show the results for models using individual feature sets in pairwise ranking models (MSE) in Table TABREF45   Language models with modified Kneser-Ney smoothing perform best on single-token hashtags, while Good-Turing and Linguistic features work best on multi-token hashtags, confirming our intuition about their usefulness in a multi-task learning approach  Table TABREF47 shows a qualitative analysis with the first column ( INLINEFORM0 INLINEFORM1 INLINEFORM2 ) indicating which features lead to correct or wrong segmentations, their count in our data and illustrative examples with human segmentation 
As expected, longer hashtags with more than three tokens pose greater challenges and the segmentation-level accuracy of our best model (MSE+multitask) drops to 82 1%  For many error cases, our model predicts a close-to-correct segmentation, e g , #youbrownknowyoubrownupttoobrownearly, #iseebrownlondoniseebrownfrance, which is also reflected by the higher token-level F INLINEFORM0 scores across hashtags with different lengths (Figure FIGREF51 ) 
Since our approach heavily relies on building a Twitter language model, we experimented with its sizes and show the results in Figure FIGREF52   Our approach can perform well even with access to a smaller amount of tweets  The drop in F INLINEFORM0 score for our pairwise neural ranker is only 1 4% and 3 9% when using the language models trained on 10% and 1% of the total 1 1 billion tweets, respectively 
Language use in Twitter changes with time BIBREF9   Our pairwise ranker uses language models trained on the tweets from the year 2010  We tested our approach on a set of 500 random English hashtags posted in tweets from the year 2019 and show the results in Table TABREF55   With a segmentation-level accuracy of 94 6% and average token-level F INLINEFORM0 score of 95 6%, our approach performs favorably on 2019 hashtags 
Extrinsic Evaluation: Twitter Sentiment Analysis
We attempt to demonstrate the effectiveness of our hashtag segmentation system by studying its impact on the task of sentiment analysis in Twitter BIBREF39 , BIBREF40 , BIBREF41   We use our best model (MSE+multitask), under the name HashtagMaster, in the following experiments 
Experimental Setup
We compare the performance of the BiLSTM+Lex BIBREF42 sentiment analysis model under three configurations: (a) tweets with hashtags removed, (b) tweets with hashtags as single tokens excluding the # symbol, and (c) tweets with hashtags as segmented by our system, HashtagMaster  BiLSTM+Lex is a state-of-the-art open source system for predicting tweet-level sentiment BIBREF43   It learns a context-sensitive sentiment intensity score by leveraging a Twitter-based sentiment lexicon BIBREF44   We use the same settings as described by BIBREF42 teng-vo-zhang:2016:EMNLP2016 to train the model 
We use the dataset from the Sentiment Analysis in Twitter shared task (subtask A) at SemEval 2017 BIBREF41   Given a tweet, the goal is to predict whether it expresses POSITIVE, NEGATIVE or NEUTRAL sentiment  The training and development sets consist of 49,669 tweets and we use 40,000 for training and the rest for development  There are a total of 12,284 tweets containing 12,128 hashtags in the SemEval 2017 test set, and our hashtag segmenter ended up splitting 6,975 of those hashtags present in 3,384 tweets 
Results and Analysis
In Table TABREF59 , we report the results based on the 3,384 tweets where HashtagMaster predicted a split, as for the rest of tweets in the test set, the hashtag segmenter would neither improve nor worsen the sentiment prediction  Our hashtag segmenter successfully improved the sentiment analysis performance by 2% on average recall and F INLINEFORM0 comparing to having hashtags unsegmented  This improvement is seemingly small but decidedly important for tweets where sentiment-related information is embedded in multi-word hashtags and sentiment prediction would be incorrect based only on the text (see Table TABREF60 for examples)  In fact, 2,605 out of the 3,384 tweets have multi-word hashtags that contain words in the Twitter-based sentiment lexicon BIBREF44 and 125 tweets contain sentiment words only in the hashtags but not in the rest of the tweet  On the entire test set of 12,284 tweets, the increase in the average recall is 0 5% 
Other Related Work
Automatic hashtag segmentation can improve the performance of many applications besides sentiment analysis, such as text classification BIBREF13 , named entity linking BIBREF10 and modeling user interests for recommendations BIBREF45   It can also help in collecting data of higher volume and quality by providing a more nuanced interpretation of its content, as shown for emotion analysis BIBREF46 , sarcasm and irony detection BIBREF11 , BIBREF47   Better semantic analysis of hashtags can also potentially be applied to hashtag annotation BIBREF48 , to improve distant supervision labels in training classifiers for tasks such as sarcasm BIBREF5 , sentiment BIBREF4 , emotions BIBREF3 ; and, more generally, as labels for pre-training representations of words BIBREF49 , sentences BIBREF50 , and images BIBREF51  
Conclusion
We proposed a new pairwise neural ranking model for hashtag segmention and showed significant performance improvements over the state-of-the-art  We also constructed a larger and more curated dataset for analyzing and benchmarking hashtag segmentation methods  We demonstrated that hashtag segmentation helps with downstream tasks such as sentiment analysis  Although we focused on English hashtags, our pairwise ranking approach is language-independent and we intend to extend our toolkit to languages other than English as future work 
Acknowledgments
We thank Ohio Supercomputer Center BIBREF52 for computing resources and the NVIDIA for providing GPU hardware  We thank Alan Ritter, Quanze Chen, Wang Ling, Pravar Mahajan, and Dushyanta Dhyani for valuable discussions  We also thank the annotators: Sarah Flanagan, Kaushik Mani, and Aswathnarayan Radhakrishnan  This material is based in part on research sponsored by the NSF under grants IIS-1822754 and IIS-1755898, DARPA through the ARO under agreement number W911NF-17-C-0095, through a Figure-Eight (CrowdFlower) AI for Everyone Award and a Criteo Faculty Research Award to Wei Xu  The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of the U S  Government 
Word-shape rules
Our model uses the following word shape rules as boolean features  If the candidate segmentation INLINEFORM0 and its corresponding hashtag INLINEFORM1 satisfies a word shape rule, then the boolean feature is set to True 
Context 2: Introduction
Emotion detection has long been a topic of interest to scholars in natural language processing (NLP) domain  Researchers aim to recognize the emotion behind the text and distribute similar ones into the same group  Establishing an emotion classifier can not only understand each user's feeling but also be extended to various application, for example, the motivation behind a user's interests BIBREF0  Based on releasing of large text corpus on social media and the emotion categories proposed by BIBREF1, BIBREF2, numerous models have provided and achieved fabulous precision so far  For example, DeepMoji BIBREF3 which utilized transfer learning concept to enhance emotions and sarcasm understanding behind the target sentence  CARER BIBREF4 learned contextualized affect representations to make itself more sensitive to rare words and the scenario behind the texts 
As methods become mature, text-based emotion detecting applications can be extended from a single utterance to a dialogue contributed by a series of utterances  Table TABREF2 illustrates the difference between single utterance and dialogue emotion recognition  The same utterances in Table TABREF2, even the same person said the same sentence, the emotion it convey may be various, which may depend on different background of the conversation, tone of speaking or personality  Therefore, for emotion detection, the information from preceding utterances in a conversation is relatively critical 
In SocialNLP 2019 EmotionX, the challenge is to recognize emotions for all utterances in EmotionLines dataset, a dataset consists of dialogues  According to the needs for considering context at the same time, we develop two classification models, inspired by bidirectional encoder representations from transformers (BERT) BIBREF5, FriendsBERT and ChatBERT  In this paper, we introduce our approaches including causal utterance modeling, model pre-training, and fine-turning 
Dataset
EmotionLines BIBREF6 is a dialogue dataset composed of two subsets, Friends and EmotionPush, according to the source of the dialogues  The former comes from the scripts of the Friends TV sitcom  The other is made up of Facebook messenger chats  Each subset includes $1,000$ English dialogues, and each dialogue can be further divided into a few consecutive utterances  All the utterances are annotated by five annotators on a crowd-sourcing platform (Amazon Mechanical Turk), and the labeling work is only based on the textual content  Annotator votes for one of the seven emotions, namely Ekman’s six basic emotions BIBREF1, plus the neutral  If none of the emotion gets more than three votes, the utterance will be marked as “non-neutral” 
For the datasets, there are properties worth additional mentioning  Although Friends and EmotionPush share the same data format, they are quite different in nature  Friends is a speech-based dataset which is annotated dialogues from the TV sitcom  It means most of the utterances are generated by the a few main characters  The personality of a character often affects the way of speaking, and therefore “who is the speaker" might provide extra clues for emotion prediction  In contrast, EmotionPush does not have this trait due to the anonymous mechanism  In addition, features such as typo, hyperlink, and emoji that only appear in chat-based data will need some domain-specific techniques to process 
Incidentally, the objective of the challenge is to predict the emotion for each utterance  Just, according to EmotionX 2019 specification, there are only four emotions be selected as our label candidates, which are Joy, Sadness, Anger, and Neutral  These emotions will be considered during performance evaluation  The technical detail will also be introduced and discussed in following Section SECREF13 and Section SECREF26 
Model Description
For this challenge, we adapt BERT which is proposed by BIBREF5 to help understand the context at the same time  Technically, BERT, designed on end-to-end architecture, is a deep pre-trained transformer encoder that dynamically provides language representation and BERT already achieved multiple state-of-the-art results on GLUE benchmark BIBREF7 and many tasks  A quick recap for BERT's architecture and its pre-training tasks will be illustrated in the following subsections 
Model Description ::: Model Architecture
BERT, the Bidirectional Encoder Representations from Transformers, consists of several transformer encoder layers that enable the model to extract very deep language features on both token-level and sentence-level  Each transformer encoder contains multi-head self-attention layers that provide ability to learn multiple attention feature of each word from their bidirectional context  The transformer and its self-attention mechanism are proposed by BIBREF8  This self-attention mechanism can be interpreted as a key-value mapping given query  By given the embedding vector for token input, the query ($Q$), key ($K$) and value ($V$) are produced by the projection from each three parameter matrices where $W^Q \in \mathbb {R}^{d_{{\rm model}} \times d_{k}}, W^K \in \mathbb {R}^{d_{\rm model} \times d_{k}}$ and $W^V \in \mathbb {R}^{d_{\rm model} \times d_{v}}$  The self-attention BIBREF8 is formally represented as:
The $ d_k = d_v = d_{\rm model} = 1024$ in BERT large version and 768 in BERT base version  Once model can extract attention feature, we can extend one self-attention into multi-head self-attention, this extension makes sub-space features can be extracted in same time by this multi-head configuration  Overall, the multi-attention mechanism is adopt for each transformer encoder, and several of encoder layer will be stacked together to form a deep transformer encoder 
For the model input, BERT allow us take one sentence as input sequence or two sentences together as one input sequence, and the maximum length of input sequence is 512  The way that BERT was designed is for giving model the sentence-level and token-level understanding  In two sentences case, a special token ([SEP]) will be inserted between two sentences  In addition, the first input token is also a special token ([CLS]), and its corresponding ouput will be vector place for classification during fine-tuning  The outputs of the last encoder layer corresponding to each input token can be treated as word representations for each token, and the word representation of the first token ([CLS]) will be consider as classification (output) representation for further fine-tuning tasks  In BERT, this vector is denoted as $C \in \mathbb {R}^{d_{\rm model}} $, and a classification layer is denoted as $ W \in \mathbb {R}^{K \times d_{\rm model}}$, where $K$ is number of classification labels  Finally, the prediction $P$ of BERT is represented as $P = {\rm softmax}(CW^T)$ 
Model Description ::: Pre-training Tasks
In pre-training, intead of using unidirectional language models, BERT developed two pre-training tasks: (1) Masked LM (cloze test) and (2) Next Sentence Prediction  At the first pre-training task, bidirectional language modeling can be done at this cloze-like pre-training  In detail, 15% tokens of input sequence will be masked at random and model need to predict those masked tokens  The encoder will try to learn contextual representations from every given tokens due to masking tokens at random  Model will not know which part of the input is going to be masked, so that the information of each masked tokens should be inferred by remaining tokens  At Next Sentence Prediction, two sentences concatenated together will be considered as model input  In order to give model a good nature language understanding, knowing relationship between sentence is one of important abilities  When generating input sequences, 50% of time the sentence B is actually followed by sentence A, and rest 50% of the time the sentence B will be picked randomly from dataset, and model need to predict if the sentence B is next sentence of sentence A  That is, the attention information will be shared between sentences  Such sentence-level understanding may have difficulties to be learned at first pre-training task (Masked LM), therefore, the pre-training task (NSP) is developed as second training goal to capture the cross sentence relationship 
In this competition, limited by the size of dataset and the challenge in contextual emotion recognition, we consider BERT with both two pre-training tasks can give a good starting point to extract emotion changing during dialogue-like conversation  Especially the second pre-training task, it might be more important for dialogue-like conversation where the emotion may various by the context of continuous utterances  That is, given a set of continues conversations, the emotion of current utterance might be influenced by previous utterance  By this assumption and with supporting from the experiment results of BERT, we can take sentence A as one-sentence context and consider sentence B as the target sentence for emotion prediction  The detail will be described in Section SECREF4 
Methodology
The main goal of the present work is to predict the emotion of utterance within the dialogue  Following are four major difficulties we concern about:
The emotion of the utterances depends not only on the text but also on the interaction happened earlier 
The source of the two datasets are different  Friends is speech-based dialogues and EmotionPush is chat-based dialogues  It makes datasets possess different characteristics 
There are only $1,000$ dialogues in both training datasets which are not large enough for the stability of training a complex neural-based model 
The prediction targets (emotion labels) are highly unbalanced 
The proposed approach is summarized in Figure FIGREF3, which aims to overcome these challenges  The framework could be separated into three steps and described as follow:
Methodology ::: Causal Utterance Modeling
Given a dialogue $D^{(i)}$ which includes sequence of utterances denoted as $D^{(i)}=(u^{(i)}_{1}, u^{(i)}_{2},    , u^{(i)}_{n})$, where $i$ is the index in dataset and $n$ is the number of utterances in the given dialogue  In order to conserve the emotional information of both utterance and conversation, we rearrange each two consecutive utterances $u_{t}, u_{t-1}$ into a single sentence representation $x_{t}$ as
The corresponding sentence representation corpus $X^{(i)}$ are denoted as $X^{(i)}=(x^{(i)}_{1}, x^{(i)}_{2},    , x^{(i)}_{n})$  Note that the first utterance within a conversation does not have its causal utterance (previous sentence), therefore, the causal utterance will be set as [None]  A practical example of sentence representation is shown in Table TABREF11 
Since the characteristics of two datasets are not identical, we customize different causal utterance modeling strategies to refine the information in text 
For Friends, there are two specific properties  The first one is that most dialogues are surrounding with the six main characters, including Rachel, Monica, Phoebe, Joey, Chandler, and Ross  The utterance ratio of given by the six roles is up to $83 4\%$  Second, the personal characteristics of the six characters are very clear  Each leading role has its own emotion undulated rule  To make use of these features, we introduce the personality tokenization which help learning the personality of the six characters  Personality tokenization concatenate the speaker and says tokens before the input utterance if the speaker is one of the six characters  The example is shown in Table TABREF12 
For EmotionPush, the text are informal chats which including like slang, acronym, typo, hyperlink, and emoji  Another characteristic is that the specific name entities are tokenized with random index  (e g  “organization_80”, “person_01”, and “time_12”)  We consider some of these informal text are related to expressing emotion such as repeated typing, purposed capitalization, and emoji (e g  “:D”, “:(”, and “<3”))  Therefore, we keep most informal expressions but only process hyperlinks, empty utterance, and name entities by unifying the tokens 
Methodology ::: Model Pre-training
Since the size of both datasets are not large enough for complex neural-based model training as well as BERT model is only pre-train on formal text datasets, the issues of overfitting and domain bias are important considerations for design the pre-training process 
To avoid our model overfitting on the training data and increase the understanding of informal text, we adapted BERT and derived two models, namely FriendsBERT and ChatBERT, with different pre-training tasks before the formal training process for Friends and EmotionPush dataset, respectively  The pre-training strategies are described below 
For pre-training FriendsBERT, we collect the completed scripts of all ten seasons of Friends TV shows from emorynlp which includes 3,107 scenes within 61,309 utterances  All the utterances are followed the preprocessing methods mentions above to compose the corpus for Masked language model pre-training task  The consequent utterances in the same scenes are considered as the consequent sentences to pre-train the Next Sentence Prediction task  In the pre-training process, the training loss is the sum of the mean likelihood of two pre-train tasks 
For pre-training ChatBERT, we pre-train our model on the Twitter dataset, since the text and writing style on Twitter are close to the chat text where both may involved with many informal words or emoticons as well  The Twitter emotion dataset, 8 basic emotions from emotion wheel BIBREF1, was collected by twitter streaming API with specific emotion-related hashtags, such as #anger, #joy, #cry, #sad and etc  The hashtags in tweets are treated as emotion label for model fine-tuning  The tweets were fine-grined processing followed the rules in BIBREF9, BIBREF4, including duplicate tweets removing, the emotion hashtags must appearing in the last position of a tweet, and etc  The statis of tweets were summarized in Table TABREF17  Each tweet and corresponding emotion label composes an emotion classification dataset for pre-training 
Methodology ::: Fine-tuning
Since our emotion recognition task is treated as a sequence-level classification task, the model would be fine-tuned on the processed training data  Following the BERT construction, we take the first embedding vector which corresponds to the special token [CLS] from the final hidden state of the Transformer encoder  This vector represents the embedding vector of the corresponding conversation utterances which is denoted as $\mathbf {C} \in \mathbb {R}^{H}$, where $H$ is the embedding size  A dense neural layer is treated as a classification layer which consists of parameters $\mathbf {W} \in \mathbb {R}^{K\times H}$ and $\mathbf {b} \in \mathbb {R}^{K}$, where $K$ is the number of emotion class  The emotion prediction probabilities $\mathbf {P} \in \mathbb {R}^{K}$ are computed by a softmax activation function as
All the parameters in BERT and the classification layer would be fine-turned together to minimize the Negative Log Likelihood (NLL) loss function, as Equation (DISPLAY_FORM22), based on the ground truth emotion label $c$ 
In order to tackle the problem of highly unbalanced emotion labels, we apply weighted balanced warming on NLL loss function, as Equation (DISPLAY_FORM23), in the first epoch of fine-tuning procedure 
where $\mathbf {w}$ are the weights of corresponding emotion label $c$ which are computed and normalize by the frequency as
By adding the weighted balanced warming on NLL loss, the model could learn to predict the minor emotions (e g  anger and sadness) earlier and make the training process more stable  Since the major evaluation metrics micro F1-score is effect by the number of each label, we only apply the weighted balanced warming in first epoch to optimize the performance 
Experiments
Since the EmotionX challenge only provided the gold labels in training data, we pick the best performance model (weights) to predict the testing data  In this section, we present the experiment and evaluation results 
Experiments ::: Experimental Setup
The EmotionX challenge consists of $1,000$ dialogues for both Friends and EmotionPush  In all of our experiments, each dataset is separated into top 800 dialogues for training and last 200 dialogues for validation  Since the EmotionX challenge considers only the four emotions (anger, joy, neutral, and sadness) in the evaluation stage, we ignore all the data point corresponding to other emotions directly  The details of emotions distribution are shown in Table TABREF18 
The hyperparameters and training setup of our models (FriendsBERT and ChatBERT) are shown in Table TABREF25  Some common and easily implemented methods are selected as the baselines embedding methods and classification models  The baseline embedding methods are including bag-of-words (BOW), term frequency–inverse document frequency (TFIDF), and neural-based word embedding  The classification models are including Logistic Regression (LR), Random Forest (RF), TextCNN BIBREF10 with initial word embedding as GloVe BIBREF11, and our proposed model  All the experiment results are based on the best performances of validation results 
Experiments ::: Performance
The experiment results of validation on Friends are shown in Table TABREF19  The proposed model and baselines are evaluated based on the Precision (P ), Recall (R ), and F1-measure (F1) 
For the traditional baselines, namely BOW and TFIDF, we observe that they achieve surprising high F1 scores around $0 81$, however, the scores for Anger and Sadness are lower  This explains that traditional approaches tend to predict the labels with large sample size, such as Joy and Neutral, but fail to take of scarce samples even when an ensemble random forest classifier is adopted  In order to prevent the unbalanced learning, we choose the weighted loss mechanism for both TextCNN and causal modeling TextCNN (C-TextCNN), these models suffer less than the traditional baselines and achieve a slightly balance performance, where there are around 15% and 7% improvement on Anger and Sadness, respectively  We following adopt the casual utterance modeling to original TextCNN, mapping previous utterance as well as target utterance into model  The causal utterance modeling improve the C-TextCNN over TextCNN for 6%, 2% and 1% on Anger, Joy and overall F1 score  Motivated from these preliminary experiments, the proposed FriendsBERT also adopt the ideas of both weighted loss and causal utterance modeling  As compared to the original BERT, single sentence BERT (FriendsBERT-base-s), the proposed FriendsBERT-base improve 1% for Joy and overall F1, and 2% for Sadness  For the final validation performance, our proposed approach achieves the highest scores, which are $0 85$ and $0 86$ for FriendsBERT-base and FriendsBERT-large, respectively 
Overall, the proposed FriendsBERT successfully captures the sentence-level context-awarded information and outperforms all the baselines, which not only achieves high performance on large sample labels, but also on small sample labels  The similar settings are also adapted to EmotionPush dataset for the final evaluation 
Experiments ::: Evaluation Results
The testing dataset consists of 240 dialogues including $3,296$ and $3,536$ utterances in Friends and EmotionPush respectively  We re-train our FriendsBERT and ChatBERT with top 920 training dialogues and predict the evaluation results using the model performing the best validation results  The results are shown in Table TABREF29 and Table TABREF30  The present method achieves $81 5\%$ and $88 5\%$ micro F1-score on the testing dataset of Friends and EmotionPush, respectively 
Conclusion and Future work
In the present work, we propose FriendsBERT and ChatBERT for the multi-utterance emotion recognition task on EmotionLines dataset  The proposed models are adapted from BERT BIBREF5 with three main improvement during the model training procedure, which are the causal utterance modeling mechanism, specific model pre-training, and adapt weighted loss  The causal utterance modeling takes the advantages of the sentence-level context information during model inference  The specific model pre-training helps to against the bias in different text domain  The weighted loss avoids our model to only predict on large size sample  The effectiveness and generalizability of the proposed methods are demonstrated from the experiments 
In future work, we consider to include the conditional probabilistic constraint $P ({\rm Emo}_{B} | \hat{\rm Emo}_{A})$  Model should predict the emotion based on a certain understanding about context emotions  This might be more reasonable for guiding model than just predicting emotion of ${\rm Sentence}_B$ directly  In addition, due to the limitation of BERT input format, ambiguous number of input sentences is now becoming an important design requirement for our future work  Also, personality embedding development will be another future work of the emotion recognition  The personality embedding will be considered as sentence embedding injected into word embedding, and it seems this additional information can contribute some improvement potentially 

ground_truth: 1,268 randomly selected tweets in the Stanford Sentiment Analysis Dataset BIBREF36, all 12,594 unique English hashtags and their associated tweets from the same Stanford dataset

Stanford Sentiment Analysis Dataset BIBREF36

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the sources of the datasets?
Context 3: Introduction
We understand from Zipf's Law that in any natural language corpus a majority of the vocabulary word types will either be absent or occur in low frequency  Estimating the statistical properties of these rare word types is naturally a difficult task  This is analogous to the curse of dimensionality when we deal with sequences of tokens - most sequences will occur only once in the training data  Neural network architectures overcome this problem by defining non-linear compositional models over vector space representations of tokens and hence assign non-zero probability even to sequences not seen during training BIBREF0 , BIBREF1   In this work, we explore a similar approach to learning distributed representations of social media posts by composing them from their constituent characters, with the goal of generalizing to out-of-vocabulary words as well as sequences at test time 
Traditional Neural Network Language Models (NNLMs) treat words as the basic units of language and assign independent vectors to each word type  To constrain memory requirements, the vocabulary size is fixed before-hand; therefore, rare and out-of-vocabulary words are all grouped together under a common type `UNKNOWN'  This choice is motivated by the assumption of arbitrariness in language, which means that surface forms of words have little to do with their semantic roles  Recently, BIBREF2 challenge this assumption and present a bidirectional Long Short Term Memory (LSTM) BIBREF3 for composing word vectors from their constituent characters which can memorize the arbitrary aspects of word orthography as well as generalize to rare and out-of-vocabulary words 
Encouraged by their findings, we extend their approach to a much larger unicode character set, and model long sequences of text as functions of their constituent characters (including white-space)  We focus on social media posts from the website Twitter, which are an excellent testing ground for character based models due to the noisy nature of text  Heavy use of slang and abundant misspellings means that there are many orthographically and semantically similar tokens, and special characters such as emojis are also immensely popular and carry useful semantic information  In our moderately sized training dataset of 2 million tweets, there were about 0 92 million unique word types  It would be expensive to capture all these phenomena in a word based model in terms of both the memory requirement (for the increased vocabulary) and the amount of training data required for effective learning  Additional benefits of the character based approach include language independence of the methods, and no requirement of NLP preprocessing such as word-segmentation 
A crucial step in learning good text representations is to choose an appropriate objective function to optimize  Unsupervised approaches attempt to reconstruct the original text from its latent representation BIBREF4 , BIBREF0   Social media posts however, come with their own form of supervision annotated by millions of users, in the form of hashtags which link posts about the same topic together  A natural assumption is that the posts with the same hashtags should have embeddings which are close to each other  Hence, we formulate our training objective to maximize cross-entropy loss at the task of predicting hashtags for a post from its latent representation 
We propose a Bi-directional Gated Recurrent Unit (Bi-GRU) BIBREF5 neural network for learning tweet representations  Treating white-space as a special character itself, the model does a forward and backward pass over the entire sequence, and the final GRU states are linearly combined to get the tweet embedding  Posterior probabilities over hashtags are computed by projecting this embedding to a softmax output layer  Compared to a word-level baseline this model shows improved performance at predicting hashtags for a held-out set of posts  Inspired by recent work in learning vector space text representations, we name our model tweet2vec 
Related Work
Using neural networks to learn distributed representations of words dates back to BIBREF0   More recently, BIBREF4 released word2vec - a collection of word vectors trained using a recurrent neural network  These word vectors are in widespread use in the NLP community, and the original work has since been extended to sentences BIBREF1 , documents and paragraphs BIBREF6 , topics BIBREF7 and queries BIBREF8   All these methods require storing an extremely large table of vectors for all word types and cannot be easily generalized to unseen words at test time BIBREF2   They also require preprocessing to find word boundaries which is non-trivial for a social network domain like Twitter 
In BIBREF2 , the authors present a compositional character model based on bidirectional LSTMs as a potential solution to these problems  A major benefit of this approach is that large word lookup tables can be compacted into character lookup tables and the compositional model scales to large data sets better than other state-of-the-art approaches  While BIBREF2 generate word embeddings from character representations, we propose to generate vector representations of entire tweets from characters in our tweet2vec model 
Our work adds to the growing body of work showing the applicability of character models for a variety of NLP tasks such as Named Entity Recognition BIBREF9 , POS tagging BIBREF10 , text classification BIBREF11 and language modeling BIBREF12 , BIBREF13  
Previously, BIBREF14 dealt with the problem of estimating rare word representations by building them from their constituent morphemes  While they show improved performance over word-based models, their approach requires a morpheme parser for preprocessing which may not perform well on noisy text like Twitter  Also the space of all morphemes, though smaller than the space of all words, is still large enough that modelling all morphemes is impractical 
Hashtag prediction for social media has been addressed earlier, for example in BIBREF15 , BIBREF16   BIBREF15 also use a neural architecture, but compose text embeddings from a lookup table of words  They also show that the learned embeddings can generalize to an unrelated task of document recommendation, justifying the use of hashtags as supervision for learning text representations 
Tweet2Vec
Bi-GRU Encoder: Figure 1 shows our model for encoding tweets  It uses a similar structure to the C2W model in BIBREF2 , with LSTM units replaced with GRU units 
The input to the network is defined by an alphabet of characters $C$ (this may include the entire unicode character set)  The input tweet is broken into a stream of characters $c_1, c_2,     c_m$ each of which is represented by a 1-by- $|C|$ encoding  These one-hot vectors are then projected to a character space by multiplying with the matrix $P_C \in \mathbb {R}^{|C| \times d_c}$ , where $d_c$ is the dimension of the character vector space  Let $x_1, x_2,     x_m$ be the sequence of character vectors for the input tweet after the lookup  The encoder consists of a forward-GRU and a backward-GRU  Both have the same architecture, except the backward-GRU processes the sequence in reverse order  Each of the GRU units process these vectors sequentially, and starting with the initial state $h_0$ compute the sequence $h_1, h_2,     h_m$ as follows: $ r_t &= \sigma (W_r x_t + U_r h_{t-1} + b_r), \\ z_t &= \sigma (W_z x_t + U_z h_{t-1} + b_z), \\ \tilde{h}_t &= tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h), \\ h_t &= (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t  $
Here $r_t$ , $z_t$ are called the reset and update gates respectively, and $\tilde{h}_t$ is the candidate output state which is converted to the actual output state $h_t$   $W_r, W_z, W_h$ are $d_h \times d_c$ matrices and $U_r, U_z, U_h$ are $d_h \times d_h$ matrices, where $d_h$ is the hidden state dimension of the GRU  The final states $h_m^f$ from the forward-GRU, and $z_t$0 from the backward GRU are combined using a fully-connected layer to the give the final tweet embedding $z_t$1 :
$$e_t = W^f h_m^f + W^b h_0^b$$   (Eq  3)
Here $W^f, W^b$ are $d_t \times d_h$ and $b$ is $d_t \times 1$ bias term, where $d_t$ is the dimension of the final tweet embedding  In our experiments we set $d_t=d_h$ 

ground_truth: Friends TV sitcom, Facebook messenger chats

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What other tasks do they test their method on?
Context 3:  All parameters are learned using gradient descent 
Softmax: Finally, the tweet embedding is passed through a linear layer whose output is the same size as the number of hashtags $L$ in the data set  We use a softmax layer to compute the posterior hashtag probabilities:
$$P(y=j |e) = \frac{exp(w_j^Te + b_j)}{\sum _{i=1}^L exp(w_i^Te + b_j)} $$   (Eq  4)
Objective Function: We optimize the categorical cross-entropy loss between predicted and true hashtags:
$$J = \frac{1}{B} \sum _{i=1}^{B} \sum _{j=1}^{L} -t_{i,j}log(p_{i,j}) + \lambda \Vert \Theta \Vert ^2 $$   (Eq  5)
Here $B$ is the batch size, $L$ is the number of classes, $p_{i,j}$ is the predicted probability that the $i$ -th tweet has hashtag $j$ , and $t_{i,j} \in \lbrace 0,1\rbrace $ denotes the ground truth of whether the $j$ -th hashtag is in the $i$ -th tweet  We use L2-regularization weighted by $\lambda $  
Word Level Baseline
Since our objective is to compare character-based and word-based approaches, we have also implemented a simple word-level encoder for tweets  The input tweet is first split into tokens along white-spaces  A more sophisticated tokenizer may be used, but for a fair comparison we wanted to keep language specific preprocessing to a minimum  The encoder is essentially the same as tweet2vec, with the input as words instead of characters  A lookup table stores word vectors for the $V$ (20K here) most common words, and the rest are grouped together under the `UNK' token 
Data
Our dataset consists of a large collection of global posts from Twitter between the dates of June 1, 2013 to June 5, 2013  Only English language posts (as detected by the lang field in Twitter API) and posts with at least one hashtag are retained  We removed infrequent hashtags ( $<500$ posts) since they do not have enough data for good generalization  We also removed very frequent tags ( $>19K$ posts) which were almost always from automatically generated posts (ex: #androidgame) which are trivial to predict  The final dataset contains 2 million tweets for training, 10K for validation and 50K for testing, with a total of 2039 distinct hashtags  We use simple regex to preprocess the post text and remove hashtags (since these are to be predicted) and HTML tags, and replace user-names and URLs with special tokens  We also removed retweets and convert the text to lower-case 
Implementation Details
Word vectors and character vectors are both set to size $d_L=150$ for their respective models  There were 2829 unique characters in the training set and we model each of these independently in a character look-up table  Embedding sizes were chosen such that each model had roughly the same number of parameters (Table 2 )  Training is performed using mini-batch gradient descent with Nesterov's momentum  We use a batch size $B=64$ , initial learning rate $\eta _0=0 01$ and momentum parameter $\mu _0=0 9$   L2-regularization with $\lambda =0 001$ was applied to all models  Initial weights were drawn from 0-mean gaussians with $\sigma =0 1$ and initial biases were set to 0  The hyperparameters were tuned one at a time keeping others fixed, and values with the lowest validation cost were chosen  The resultant combination was used to train the models until performance on validation set stopped increasing  During training, the learning rate is halved everytime the validation set precision increases by less than 0 01 % from one epoch to the next  The models converge in about 20 epochs  Code for training both the models is publicly available on github 
Results
We test the character and word-level variants by predicting hashtags for a held-out test set of posts  Since there may be more than one correct hashtag per post, we generate a ranked list of tags for each post from the output posteriors, and report average precision@1, recall@10 and mean rank of the correct hashtags  These are listed in Table 3  
To see the performance of each model on posts containing rare words (RW) and frequent words (FW) we selected two test sets each containing 2,000 posts  We populated these sets with posts which had the maximum and minimum number of out-of-vocabulary words respectively, where vocabulary is defined by the 20K most frequent words  Overall, tweet2vec outperforms the word model, doing significantly better on RW test set and comparably on FW set  This improved performance comes at the cost of increased training time (see Table 2 ), since moving from words to characters results in longer input sequences to the GRU 
We also study the effect of model size on the performance of these models  For the word model we set vocabulary size $V$ to 8K, 15K and 20K respectively  For tweet2vec we set the GRU hidden state size to 300, 400 and 500 respectively  Figure 2 shows precision 1 of the two models as the number of parameters is increased, for each test set described above  There is not much variation in the performance, and moreover tweet2vec always outperforms the word based model for the same number of parameters 
Table 4 compares the models as complexity of the task is increased  We created 3 datasets (small, medium and large) with an increasing number of hashtags to be predicted  This was done by varying the lower threshold of the minimum number of tags per post for it to be included in the dataset  Once again we observe that tweet2vec outperforms its word-based counterpart for each of the three settings 
Finally, table 1 shows some predictions from the word level model and tweet2vec  We selected these to highlight some strengths of the character based approach - it is robust to word segmentation errors and spelling mistakes, effectively interprets emojis and other special characters to make predictions, and also performs comparably to the word-based approach for in-vocabulary tokens 
Conclusion
We have presented tweet2vec - a character level encoder for social media posts trained using supervision from associated hashtags  Our result shows that tweet2vec outperforms the word based approach, doing significantly better when the input post contains many rare words  We have focused only on English language posts, but the character model requires no language specific preprocessing and can be extended to other languages  For future work, one natural extension would be to use a character-level decoder for predicting the hashtags  This will allow generation of hashtags not seen in the training dataset  Also, it will be interesting to see how our tweet2vec embeddings can be used in domains where there is a need for semantic understanding of social media, such as tracking infectious diseases BIBREF17   Hence, we provide an off-the-shelf encoder trained on medium dataset described above to compute vector-space representations of tweets along with our code on github 
Acknowledgments
We would like to thank Alex Smola, Yun Fu, Hsiao-Yu Fish Tung, Ruslan Salakhutdinov, and Barnabas Poczos for useful discussions  We would also like to thank Juergen Pfeffer for providing access to the Twitter data, and the reviewers for their comments 

ground_truth: None