prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How big is the difference in performance between proposed model and baselines?
Context 1: Introduction
Attempts toward constructing human-like dialogue agents have met significant difficulties, such as maintaining conversation consistency BIBREF0  This is largely due to inabilities of dialogue agents to engage the user emotionally because of an inconsistent personality BIBREF1  Many agents use personality models that attempt to map personality attributes into lower dimensional spaces (e g  the Big Five BIBREF2)  However, these represent human personality at a very high-level and lack depth  They prohibit the ability to link specific and detailed personality traits to characters, and to construct large datasets where dialogue is traceable back to these traits 
For this reason, we propose Human Level Attributes (HLAs), which we define as characteristics of fictional characters representative of their profile and identity  We base HLAs on tropes collected from TV Tropes BIBREF3, which are determined by viewers' impressions of the characters  See Figure FIGREF1 for an example  Based on the hypothesis that profile and identity contribute effectively to language style BIBREF4, we propose that modeling conversation with HLAs is a means for constructing a dialogue agent with stable human-like characteristics  By collecting dialogues from a variety of characters along with this HLA information, we present a novel labelling of this dialogue data where it is traceable back to both its context and associated human-like qualities 
We also propose a system called ALOHA (Artificial Learning On Human Attributes) as a novel method of incorporating HLAs into dialogue agents  ALOHA maps characters to a latent space based on their HLAs, determines which are most similar in profile and identity, and recovers language styles of specific characters  We test the performance of ALOHA in character language style recovery against four baselines, demonstrating outperformance and system stability  We also run a human evaluation supporting our results  Our major contributions are: (1) We propose HLAs as personality aspects of fictional characters from the audience's perspective based on tropes; (2) We provide a large dialogue dataset traceable back to both its context and associated human-like attributes; (3) We propose a system called ALOHA that is able to recommend responses linked to specific characters  We demonstrate that ALOHA, combined with the proposed dataset, outperforms baselines  ALOHA also shows stable performance regardless of the character's identity, genre of the show, and context of the dialogue  We plan to release all of ALOHA's data and code 
Related Work
Task completion chatbots (TCC), or task-oriented chatbots, are dialogue agents used to fulfill specific purposes, such as helping customers book airline tickets, or a government inquiry system  Examples include the AIML based chatbot BIBREF5 and DIVA Framework BIBREF6  While TCC are low cost, easily configurable, and readily available, they are restricted to working well for particular domains and tasks 
Open-domain chatbots are more generic dialogue systems  An example is the Poly-encoder from BIBREF7 humeau2019real  It outperforms the Bi-encoder BIBREF8, BIBREF9 and matches the performance of the Cross-encoder BIBREF10, BIBREF11 while maintaining reasonable computation time  It performs strongly on downstream language understanding tasks involving pairwise comparisons, and demonstrates state-of-the-art results on the ConvAI2 challenge BIBREF12  Feed Yourself BIBREF13 is an open-domain dialogue agent with a self-feeding model  When the conversation goes well, the dialogue becomes part of the training data, and when the conversation does not, the agent asks for feedback  Lastly, Kvmemnn BIBREF14 is a key-value memory network with a knowledge base that uses a key-value retrieval mechanism to train over multiple domains simultaneously  We use all three of these models as baselines for comparison  While these can handle a greater variety of tasks, they do not respond with text that aligns with particular human-like characteristics 
BIBREF15 li2016persona defines persona (composite of elements of identity) as a possible solution at the word level, using backpropagation to align responses via word embeddings  BIBREF16 bartl2017retrieval uses sentence embeddings and a retrieval model to achieve higher accuracy on dialogue context  BIBREF17 liu2019emotion applies emotion states of sentences as encodings to select appropriate responses  BIBREF18 pichl2018alquist uses knowledge aggregation and hierarchy of sub-dialogues for high user engagement  However, these agents represent personality at a high-level and lack detailed human qualities  LIGHT BIBREF19 models adventure game characters' dialogues, actions, and emotions  It focuses on the agent identities (e g  thief, king, servant) which includes limited information on realistic human behaviours  BIBREF20 pasunuru2018game models online soccer games as dynamic visual context  BIBREF21 wang2016learning models user dialogue to complete tasks involving certain configurations of blocks  BIBREF22 antol2015vqa models open-ended questions, but is limited to visual contexts  BIBREF23 bordes2016learning tracks user dialogues but is goal-oriented  BIBREF24 ilinykh2019meetup tracks players' dialogues and movements in a visual environment, and is grounded on navigation tasks  All of these perform well in their respective fictional environments, but are not a strong representation of human dialogue in reality 
Methodology ::: Human Level Attributes (HLA)
We collect HLA data from TV Tropes BIBREF3, a knowledge-based website dedicated to pop culture, containing information on a plethora of characters from a variety of sources  Similar to Wikipedia, its content is provided and edited collaboratively by a massive user-base  These attributes are determined by human viewers and their impressions of the characters, and are correlated with human-like characteristics  We believe that TV Tropes is better for our purpose of fictional character modeling than data sources used in works such as BIBREF25 shuster2019engaging because TV Tropes' content providers are rewarded for correctly providing content through community acknowledgement 
TV Tropes defines tropes as attributes of storytelling that the audience recognizes and understands  We use tropes as HLAs to calculate correlations with specific target characters  We collect data from numerous characters from a variety of TV shows, movies, and anime  We filter and keep characters with at least five HLA, as those with fewer are not complex enough to be correctly modeled due to reasons such as lack of data  We end up eliminating 5 86% of total characters, and end up with 45,821 characters and 12,815 unique HLA, resulting in 945,519 total character-HLA pairs  Each collected character has 20 64 HLAs on average  See Figure FIGREF1 for an example character and their HLAs 
Methodology ::: Overall Task
Our task is the following, where $t$ denotes “target":
Given a target character $c_t$ with HLA set $H_t$, recover the language style of $c_t$ without any dialogue of $c_t$ provided 
For example, if Sheldon Cooper from The Big Bang Theory is $c_t$, then $H_t$ is the set of HLA on the left side of Figure FIGREF1 
We define the language style of a character as its diction, tone, and speech patterns  It is a character specific language model refined from a general language model  We must learn to recover the language style of $c_t$ without its dialogue as our objective is to imitate human-like qualities, and hence the model must understand the language styles of characters based on their traits  If we feed $c_t$'s dialogue during training, the model will likely not effectively learn to imitate language styles based on HLAs, but based on the correlation between text in the training and testing dialogues BIBREF26 
We define character space as the character representations within the HLA latent space (see Figure FIGREF4), and the set $C = \lbrace c_1,c_2,   ,c_n\rbrace $ as the set of all characters  We define Observation (OBS) as the input that is fed into any dialogue model  This can be a single or multiple lines of dialogue along with other information  The goal of the dialogue model is to find the best response to this OBS 
Methodology ::: ALOHA
We propose a three-component system called ALOHA to solve the task (see Figure FIGREF6)  The first component, Character Space Module (CSM), generates the character space and calculates confidence levels using singular value decomposition BIBREF27 between characters $c_j$ (for $j = 1$ to $n$ where $j \ne t$) and $c_t$ in the HLA-oriented neighborhood 
The second component, Character Community Module (CCM), ranks the similarity between our target character $c_t$ with any other character $c_j$ by the relative distance between them in the character space 
The third component, Language Style Recovery Module (LSRM), recovers the language style of $c_t$ without its dialogue by training the BERT bi-ranker model BIBREF28 to rank responses from similar characters  Our results demonstrate higher accuracy at retrieving the ground truth response from $c_t$  Our system is also able to pick responses which are correct both in context as well as character space 
Hence, the overall process for ALOHA works as follows  First, given a set of characters, determine the character space using the CSM  Next, given a specific target character, determine the positive community and negative set of associated characters using the CCM  Lastly, using the positive community and negative set determined above along with a dialogue dataset, recover the language style of the target 
Methodology ::: Character Space Module (CSM)
CSM learns how to rank characters  We can measure the interdependencies between the HLA variables BIBREF29 and rank the similarity between the TV show characters  We use implicit feedback instead of neighborhood models (e g  cosine similarity) because it can compute latent factors to transform both characters and HLAs into the same latent space, making them directly comparable 
We define a matrix $P$ that contains binary values, with $P_{u,i} = 1$ if character $u$ has HLA $i$ in our dataset, and $P_{u,i} = 0$ otherwise  We define a constant $\alpha $ that measures our confidence in observing various character-HLA pairs as positive  $\alpha $ controls how much the model penalizes the error if the ground truth is $P_{u,i} = 1$  If $P_{u,i} = 1$ and the model guesses incorrectly, we penalize by $\alpha $ times the loss  But if $P_{u,i} = 0$ and the model guesses a value greater than 0, we do not penalize as $\alpha $ has no impact  This is because $P_{u,i} = 0$ can either represent a true negative or be due to a lack of data, and hence is less reliable for penalization  See Equation DISPLAY_FORM8  We find that using $\alpha =20$ provides decent results 
We further define two dense vectors $X_u$ and $Y_i$  We call $X_u$ the “latent factors for character $u$", and $Y_i$ the “latent factors for HLA $i$"  The dot product of these two vectors produces a value ($X_u^TY_i$) that approximates $P_{u,i}$ (see Figure FIGREF9)  This is analogous to factoring the matrix $P$ into two separate matrices, where one contains the latent factors for characters, and the other contains the latent factors for HLAs  We find that $X_u$ and $Y_i$ being 36-dimensional produces decent results  To bring $X_u^TY_i$ as close as possible to $P_{u,i}$, we minimize the following loss function using the Conjugate Gradient Method BIBREF30:
The first term penalizes differences between the model's prediction ($X_u^TY_i$) and the actual value ($P_{u,i}$)  The second term is an L2 regularizer to reduce overfitting  We find $\lambda = 100$ provides decent results for 500 iterations (see Section SECREF26) 
Methodology ::: Character Community Module (CCM)
CCM aims to divide characters (other than $c_t$) into a positive community and a negative set  We define this positive community as characters that are densely connected internally to $c_t$ within the character space, and the negative set as the remaining characters  We can then sample dialogue from characters in the negative set to act as the distractors (essentially negative samples) during LSRM training 
As community finding is an ill-defined problem BIBREF31, we choose to treat CCM as a simple undirected, unweighted graph  We use the values learned in the CSM for $X_u$ and $Y_i$ for various values of $u$ and $i$, which approximate the matrix $P$  Similar to BIBREF29 hu2008collaborative, we can calculate the correlation between two rows (and hence two characters) 
We then employ a two-level connection representation by ranking all characters against each other in terms of their correlation with $c_t$  For the first level, the set $S^{FL}$ is the top 10% (4582) most highly correlated characters with $c_t$ out of the 45,820 total other characters that we have HLA data for  For the second level, for each character $s_i$ in $S^{FL}$, we determine the 30 most heavily correlated characters with $s_i$ as set $S^{SL}_i$  The positive set $S^{pos}$ are the characters which are present in at least 10 $S^{SL}_i$ sets  We call this value 10 the minimum frequency  All other characters in our dialogue dataset make up the negative set $S^{neg}$  These act as our positive community and negative set, respectively  See Algorithm 1 in Appendix A for details, and Figure FIGREF11 for an example 
Methodology ::: Language Style Recovery Module (LSRM)
LSRM creates a dialogue agent that aligns with observed characteristics of human characters by using the positive character community and negative set determined in the CCM, along with a dialogue dataset, to recover the language style of $c_t$ without its dialogue  We use the BERT bi-ranker model from the Facebook ParlAI framework BIBREF32, where the model has the ability to retrieve the best response out of 20 candidate responses  BIBREF12, BIBREF19, BIBREF0 choose 20 candidate responses, and for comparison purposes, we do the same 
Methodology ::: Language Style Recovery Module (LSRM) ::: BERT
BIBREF28 is first trained on massive amounts of unlabeled text data  It jointly conditions on text on both the left and right, which provides a deep bi-directional representation of sentence inference  BERT is proven to perform well on a wide range of tasks by simply fine-tuning on one additional layer  We are interested in its ability to predict the next sentence, called Next Sentence Prediction  We perform further fine-tuning on BERT for our target character language style retrieval task to produce our LSRM model by optimizing both the encoding layers and the additional layer  We use BERT to create vector representations for the OBS and for each candidate response  By passing the first output of BERT's 12 layers through an additional linear layer, these representations can be obtained as 768-dimensional sentence-level embeddings  It uses the dot product between these embeddings to score candidate responses and is trained using the ranking loss 
Methodology ::: Language Style Recovery Module (LSRM) ::: Candidate response selection
is similar to the procedure from previous work done on grounded dialogue agents BIBREF0, BIBREF19  Along with the ground truth response, we randomly sample 19 distractor responses from other characters from a uniform distribution of characters, and call this process uniform character sampling  Based on our observations, this random sampling provides multiple context correct responses  Hence, the BERT bi-ranker model is trained by learning to choose context correct responses, and the model learns to recover a domain-general language model that includes training on every character  This results in a Uniform Model that can select context correct responses, but not responses corresponding to a target character with specific HLAs 
We then fine-tune on the above model to produce our LSRM model with a modification: we randomly sample the 19 distractor responses from only the negative character set instead  We choose the responses that have similar grammatical structures and semantics to the ground truth response, and call this process negative character sampling  This guides the model away from the language style of these negative characters to improve performance at retrieving responses for target characters with specific HLAs  Our results demonstrate higher accuracy at retrieving the correct response from character $c_t$, which is the ground truth 
Experiment ::: Dialogue Dataset
To train the Uniform Model and LSRM, we collect dialogues from 327 major characters (a subset of the 45,821 characters we have HLA data for) in 38 TV shows from various existing sources of clean data on the internet, resulting in a total of 1,042,647 dialogue lines  We use a setup similar to the Persona-Chat dataset BIBREF0 and Cornell Movie-Dialogs Corpus BIBREF33, as our collected dialogues are also paired in terms of valid conversations  See Figure FIGREF1 for an example of these dialogue lines 
Experiment ::: HLA Observation Guidance (HLA-OG)
We define HLA Observation Guidance (HLA-OG) as explicitly passing a small subset of the most important HLAs of a given character as part of the OBS rather than just an initial line of dialogue  This is adapted from the process used in BIBREF0 zhang2018personalizing and BIBREF10 wolf2019transfertransfo which we call Persona Profiling  Specifically, we pass four HLAs that are randomly drawn from the top 40 most important HLAs of the character  We use HLA-OG during training of the LSRM and testing of all models  This is because the baselines (see Section SECREF31) already follow a similar process (Persona Profiling) for training  For the Uniform Model, we train using Next Sentence Prediction (see Section SECREF12)  For testing, HLA-OG is necessary as it provides information about which HLAs the models should attempt to imitate in their response selection  Just passing an initial line of dialogue replicates a typical dialogue response task without HLAs  See Table TABREF19  Further, we also test our LSRM by explicitly passing four HLAs of `none' along with the initial line of dialogue as the OBS (No HLA-OG in Table TABREF19) 
Experiment ::: Training Details ::: BERT bi-ranker
is trained by us on the Persona-Chat dataset for the ConvAI2 challenge  Similar to BIBREF0 zhang2018personalizing, we cap the length of the OBS at 360 tokens and the length of each candidate response at 72 tokens  We use a batch size of 64, learning rate of 5e-5, and perform warm-up updates for 100 iterations  The learning rate scheduler uses SGD optimizer with Nesterov's accelerated gradient descent BIBREF34 and is set to have a decay of 0 4 and to reduce on plateau 
Experiment ::: Training Details ::: Uniform Model
is produced by finetuning the BERT bi-ranker on the dialogue data discussed in Section SECREF15 using uniform character sampling  We use the same hyperparameters as the BERT bi-ranker along with half-precision operations (i e  float16 operations) to increase batch size as recommended BIBREF7 
Experiment ::: Training Details ::: LSRM
is produced by finetuning on the Uniform Model discussed above using negative character sampling  We use the same hyperparameters as the BERT bi-ranker along with half-precision operations (i e  float16 operations) to increase batch size as recommended 
Evaluation ::: CSM Evaluation
We begin by evaluating the ability of the CSM component of our system to correctly generate the character space  To do so, during training, 30% of the character-HLA pairs (which are either 0 or 1) are masked, and this is used as a validation set (see Figure FIGREF9)  For each character $c$, the model generates a list of the 12,815 unique HLAs ranked similarly to BIBREF29 hu2008collaborative for $c$  We look at the recall of our CSM model, which measures the percentage of total ground truth HLAs (over all characters $c$) present within the top N ranked HLAs for all $c$ by our model  That is:
where $HLA_{c}^{gt}$ are the ground truth HLAs for $c$, and $HLA_{c}^{tN}$ are the top N ranked HLAs by the model for $c$  We use $N = 100$, and our model achieves 25 08% recall 
To inspect the CSM performance, we use the T-distributed Stochastic Neighbor Embedding (t-SNE) BIBREF35 to reduce each high-dimensionality data point to two-dimensions via Kullback-Leibler Divergence BIBREF36  This allows us to map our character space into two-dimensions, where similar characters from our embedding space have higher probability of being mapped close by  We sampled characters from four different groups or regions  As seen in Figure FIGREF4, our learned character space effectively groups these characters, as similar characters are adjacent to one another in four regions 
Evaluation ::: Automatic Evaluation Setup ::: Five-Fold Cross Validation
is used for training and testing of the Uniform Model and LSRM  The folds are divided randomly by the TV shows in our dialogue data  We use the dialogue data for 80% of these shows as the four-folds for training, and the dialogue data for the remaining 20% as the fifth-fold for validation/testing  The dialogue data used is discussed in Section SECREF15  This ensures no matter how our data is distributed, each part of it is tested, allowing our evaluation to be more robust to different characters  See Appendix C for five-fold cross validation details and statistics 
Evaluation ::: Automatic Evaluation Setup ::: Five Evaluation Characters
are chosen, one from each of the five testing sets above  Each is a well-known character from a separate TV show, and acts as a target character $c_t$ for evaluation of every model  We choose Sheldon Cooper from The Big Bang Theory, Jean-Luc Picard from Star Trek, Monica Geller from Friends, Gil Grissom from CSI, and Marge Simpson from The Simpsons  We choose characters of significantly different identities and profiles (intelligent scientist, ship captain, outgoing friend, police leader, and responsible mother, respectively) from shows of a variety of genres to ensure that we can successfully recover the language styles of various types of characters  We choose well-known characters because humans require knowledge on the characters they are evaluating (see Section SECREF40) 
For each of these five evaluation characters, all the dialogue lines from the character act as the ground truth responses  The initial dialogue lines are the corresponding dialogue lines to which these ground truth responses are responding  For each initial dialogue line, we randomly sample 19 other candidate responses from the associated testing set using uniform character sampling  Note that this is for evaluation, and hence we use the same uniform character sampling method for all models including ALOHA  The use of negative character sampling is only in ALOHA's training 
Evaluation ::: Baselines
We compare against four dialogue system baselines: Kvmemnn, Feed Yourself, Poly-encoder, and a BERT bi-ranker baseline trained on the Persona-Chat dataset using the same training hyperparameters (including learning rate scheduler and length capping settings) described in Section SECREF20  For the first three models, we use the provided pretrained (on Persona-Chat) models  We evaluate all four on our five evaluation characters discussed in Section SECREF28 
Evaluation ::: Key Evaluation Metrics ::: Hits@n/N
is the accuracy of the correct ground truth response being within the top $n$ ranked candidate responses out of $N$ total candidates  We measure Hits@1/20, Hits@5/20, and Hits@10/20 
Evaluation ::: Key Evaluation Metrics ::: Mean Rank
is the average rank that a model assigns the ground truth response among the 20 total candidates 
Evaluation ::: Key Evaluation Metrics ::: Mean Reciprocal Rank (MRR)
BIBREF37 looks at the mean of the multiplicative inverses of the rank of each correct answer out of a sample of queries $Q$:
where $rank_i$ refers to the rank position of the correct response for the $i$-th query, and $|Q|$ refers to the total number of queries in $Q$ 
Evaluation ::: Key Evaluation Metrics ::: @!START@$F_1$@!END@-score
equals $2 * \frac{precision*recall}{precision+recall}$  For dialogue, precision is the fraction of words in the chosen response contained in the ground truth response, and recall is the fraction of words in the ground truth response contained in the chosen response 
Evaluation ::: Key Evaluation Metrics ::: BLEU
BIBREF38 generally indicates how close two pieces of text are in content and structure, with higher values indicating greater similarity  We report our final BLEU scores as the average scores of 1 to 4-grams 
Evaluation ::: Human Evaluation Setup
We conduct a human evaluation with 12 participants, 8 male and 4 female, who are affiliated project researchers aged 20-39 at the University of [ANON]  We choose the same five evaluation characters as in Section SECREF28  To control bias, each participant evaluates one or two characters  For each character, we randomly select 10 testing samples (each includes an initial line of dialogue along with 20 candidate responses, one of which is the ground truth) from the same testing data for the automatic evaluation discussed in Section SECREF28 
These ten samples make up a single questionnaire presented in full to each participant evaluating the corresponding character, and the participant is asked to select the single top response they think the character would most likely respond with for each of the ten initial dialogue lines  See Figure FIGREF41 for an example  We mask any character names within the candidate responses to prevent human participants from using names to identify which show the response is from 
Each candidate is prescreened to ensure they have sufficient knowledge of the character to be a participant  We ask three prescreening questions where the participant has to identify an image, relationship, and occupation of the character  All 12 of our participants passed the the prescreening 
Results and Analysis ::: Evaluation Results
Table TABREF44 shows average results of our automatic and human evaluations  Table TABREF45 shows average Hits@1/20 scores by evaluation character  See Appendix F for detailed evaluation results  ALOHA is the model with HLA-OG during training and testing, and ALOHA (No HLA-OG) is the model with HLA-OG during training but tested with the four HLAs in the OBS marked as `none' (see Section SECREF17)  See Appendix G for demo interactions between a human, BERT bi-ranker baseline, and ALOHA for all five evaluation characters 
Results and Analysis ::: Evaluation Challenges
The evaluation of our task (retrieving the language style of a specific character) is challenging and hence the five-fold cross validation is necessary for the following reasons:
The ability to choose a context correct response without attributes of specific characters may be hard to separate from our target metric, which is the ability to retrieve the correct response of a target character by its HLAs  However, from manual observation, we noticed that in the 20 chosen candidate responses, there are typically numerous context correct responses, but only one ground truth for the target character (for an example, see Figure FIGREF41)  Hence, a model that only chooses dialogue based on context is distinguishable from one that learns HLAs 
Retrieving responses for the target character depends on the other candidate responses  For example, dialogue retrieval performance for Grissom from CSI, which is a crime/police context, is higher than other evaluation characters (see Table TABREF45), potentially due to other candidate responses not falling within the same crime/police context 
Results and Analysis ::: Performance: ALOHA vs  Humans
As observed from Table TABREF44, ALOHA has a performance relatively close to humans  Human Hits@1/20 scores have a mean of 40 67% and a median over characters of 40%  The limited human evaluation sample size limits what can be inferred, but it indicates that the problem is solved to the extent that ALOHA is able perform relatively close to humans on average  Notice that even humans do not perform extremely well, demonstrating that this task of character based dialogue retrieval is more difficult than typical dialogue retrieval tasks BIBREF19, BIBREF12 
Looking more closely at each character from Table TABREF45, we can see that human evaluation scores are higher for Sheldon and Grissom  This may be due to these characters having more distinct personalities, making them more memorable 
We also look at Pearson correlation values of the Hits@1/20 scores across the five evaluation characters  For human versus Uniform Model, this is -0 4694, demonstrating that the Uniform Model, without knowledge of HLAs, fails to imitate human impressions  For human versus ALOHA, this is 0 4250, demonstrating that our system is able to retrieve character responses somewhat similarly to human impressions  Lastly, for human versus the difference in scores between ALOHA and Uniform Model, this is 0 7815  The difference between ALOHA and the Uniform Model, which is based on the additional knowledge of the HLAs, is hence shown to improve upon the Uniform Model similarly to human impressions  This demonstrates that HLAs are indeed an accurate method of modeling human impressions of character attributes, and also demonstrates that our system, ALOHA, is able to effectively use these HLAs to improve upon dialogue retrieval performance 
Results and Analysis ::: Performance: ALOHA vs  Baselines
ALOHA, combined with the HLAs and dialogue dataset, achieves a significant improvement on the target character language style retrieval task compared to the baseline open-domain chatbot models  As observed from Table TABREF44, ALOHA achieves a significant boost in Hits@n/N accuracy and other metrics for retrieving the correct response of five diverse characters with different identities (see Section SECREF28) 
Results and Analysis ::: Performance: ALOHA vs  Uniform Model
We observe a noticeable improvement in performance between ALOHA and the Uniform Model in recovering the language styles of specific characters that is consistent across all five folds (see Tables TABREF44 and TABREF45), indicating that lack of knowledge of HLAs limits the ability of the model to successfully recover the language style of specific characters  We claim that, to the best of our knowledge, we have made the first step in using HLA-based character dialogue clustering to improve upon personality learning for chatbots 
ALOHA demonstrates an accuracy boost for all five evaluation characters, showing that the system is robust and stable and has the ability to recover the dialogue styles of fictional characters regardless of the character's profile and identity, genre of the show, and context of the dialogue 
Results and Analysis ::: Performance: HLA-OG
As observed from Table TABREF44, ALOHA performs slightly better overall compared to ALOHA (No HLA-OG)  Table TABREF45 shows that this slight performance increase is consistent across four of the five evaluation characters  In the case of Sheldon, the HLA-OG model performs a bit worse  This is possibly due to the large number of Sheldon's HLAs (217) compared to the other four evaluation characters (average of 93 75), along with the limited amount of HLAs we are using for guidance due to the models' limited memory  In general, HLA Observation Guidance during testing appears to improve upon the performance of ALOHA, but this improvement is minimal 
Conclusion and Future Work
We proposed Human Level Attributes (HLAs) as a novel approach to model human-like attributes of characters, and collected a large volume of dialogue data for various characters with complete and robust profiles  We also proposed and evaluated a system, ALOHA, that uses HLAs to recommend tailored responses traceable to specific characters, and demonstrated its outperformance of the baselines and ability to effectively recover language styles of various characters, showing promise for learning character or personality styles  ALOHA was also shown to be stable regardless of the character's identity, genre of show, and context of dialogue 
Potential directions for future work include training ALOHA with a multi-turn response approach BIBREF0 that tracks dialogue over multiple responses, as we could not acquire multi-turn dialogue data for TV shows  Another potential is the modeling of the dialog counterpart (e g  the dialogue of other characters speaking to the target character)  Further, performing semantic text exchange on the chosen response with a model such as SMERTI BIBREF39 may improve the ability of ALOHA to converse with humans  This is because the response may be context and HLA correct, but incorrect semantically (e g  the response may say the weather is sunny when it is actually rainy)  HLA-aligned generative models is another area of exploration  Typically, generative models produce text that is less fluent, but further work in this area may lead to better results  Lastly, a more diverse and larger participant pool is required due to the limited size of our human evaluation 
Context 2: Introduction
Recently, deep learning algorithms have successfully addressed problems in various fields, such as image classification, machine translation, speech recognition, text-to-speech generation and other machine learning related areas BIBREF0 , BIBREF1 , BIBREF2   Similarly, substantial improvements in performance have been obtained when deep learning algorithms have been applied to statistical speech processing BIBREF3   These fundamental improvements have led researchers to investigate additional topics related to human nature, which have long been objects of study  One such topic involves understanding human emotions and reflecting it through machine intelligence, such as emotional dialogue models BIBREF4 , BIBREF5  
In developing emotionally aware intelligence, the very first step is building robust emotion classifiers that display good performance regardless of the application; this outcome is considered to be one of the fundamental research goals in affective computing BIBREF6   In particular, the speech emotion recognition task is one of the most important problems in the field of paralinguistics  This field has recently broadened its applications, as it is a crucial factor in optimal human-computer interactions, including dialog systems  The goal of speech emotion recognition is to predict the emotional content of speech and to classify speech according to one of several labels (i e , happy, sad, neutral, and angry)  Various types of deep learning methods have been applied to increase the performance of emotion classifiers; however, this task is still considered to be challenging for several reasons  First, insufficient data for training complex neural network-based models are available, due to the costs associated with human involvement  Second, the characteristics of emotions must be learned from low-level speech signals  Feature-based models display limited skills when applied to this problem 
To overcome these limitations, we propose a model that uses high-level text transcription, as well as low-level audio signals, to utilize the information contained within low-resource datasets to a greater degree  Given recent improvements in automatic speech recognition (ASR) technology BIBREF7 , BIBREF2 , BIBREF8 , BIBREF9 , speech transcription can be carried out using audio signals with considerable skill  The emotional content of speech is clearly indicated by the emotion words contained in a sentence BIBREF10 , such as “lovely” and “awesome,” which carry strong emotions compared to generic (non-emotion) words, such as “person” and “day ” Thus, we hypothesize that the speech emotion recognition model will be benefit from the incorporation of high-level textual input 
In this paper, we propose a novel deep dual recurrent encoder model that simultaneously utilizes audio and text data in recognizing emotions from speech  Extensive experiments are conducted to investigate the efficacy and properties of the proposed model  Our proposed model outperforms previous state-of-the-art methods by 68 8% to 71 8% when applied to the IEMOCAP dataset, which is one of the most well-studied datasets  Based on an error analysis of the models, we show that our proposed model accurately identifies emotion classes  Moreover, the neutral class misclassification bias frequently exhibited by previous models, which focus on audio features, is less pronounced in our model 
Related work
Classical machine learning algorithms, such as hidden Markov models (HMMs), support vector machines (SVMs), and decision tree-based methods, have been employed in speech emotion recognition problems BIBREF11 , BIBREF12 , BIBREF13   Recently, researchers have proposed various neural network-based architectures to improve the performance of speech emotion recognition  An initial study utilized deep neural networks (DNNs) to extract high-level features from raw audio data and demonstrated its effectiveness in speech emotion recognition BIBREF14   With the advancement of deep learning methods, more complex neural-based architectures have been proposed  Convolutional neural network (CNN)-based models have been trained on information derived from raw audio signals using spectrograms or audio features such as Mel-frequency cepstral coefficients (MFCCs) and low-level descriptors (LLDs) BIBREF15 , BIBREF16 , BIBREF17   These neural network-based models are combined to produce higher-complexity models BIBREF18 , BIBREF19 , and these models achieved the best-recorded performance when applied to the IEMOCAP dataset 
Another line of research has focused on adopting variant machine learning techniques combined with neural network-based models  One researcher utilized the multiobject learning approach and used gender and naturalness as auxiliary tasks so that the neural network-based model learned more features from a given dataset BIBREF20   Another researcher investigated transfer learning methods, leveraging external data from related domains BIBREF21  
As emotional dialogue is composed of sound and spoken content, researchers have also investigated the combination of acoustic features and language information, built belief network-based methods of identifying emotional key phrases, and assessed the emotional salience of verbal cues from both phoneme sequences and words BIBREF22 , BIBREF23   However, none of these studies have utilized information from speech signals and text sequences simultaneously in an end-to-end learning neural network-based model to classify emotions 
Model
This section describes the methodologies that are applied to the speech emotion recognition task  We start by introducing the recurrent encoder model for the audio and text modalities individually  We then propose a multimodal approach that encodes both audio and textual information simultaneously via a dual recurrent encoder 
Audio Recurrent Encoder (ARE)
Motivated by the architecture used in BIBREF24 , BIBREF25 , we build an audio recurrent encoder (ARE) to predict the class of a given audio signal  Once MFCC features have been extracted from an audio signal, a subset of the sequential features is fed into the RNN (i e , gated recurrent units (GRUs)), which leads to the formation of the network's internal hidden state INLINEFORM0 to model the time series patterns  This internal hidden state is updated at each time step with the input data INLINEFORM1 and the hidden state of the previous time step INLINEFORM2 as follows: DISPLAYFORM0
where INLINEFORM0 is the RNN function with weight parameter INLINEFORM1 , INLINEFORM2 represents the hidden state at t- INLINEFORM3 time step, and INLINEFORM4 represents the t- INLINEFORM5 MFCC features in INLINEFORM6   After encoding the audio signal INLINEFORM7 with the RNN, the last hidden state of the RNN, INLINEFORM8 , is considered to be the representative vector that contains all of the sequential audio data  This vector is then concatenated with another prosodic feature vector, INLINEFORM9 , to generate a more informative vector representation of the signal, INLINEFORM10   The MFCC and the prosodic features are extracted from the audio signal using the openSMILE toolkit BIBREF26 , INLINEFORM11 , respectively  Finally, the emotion class is predicted by applying the softmax function to the vector INLINEFORM12   For a given audio sample INLINEFORM13 , we assume that INLINEFORM14 is the true label vector, which contains all zeros but contains a one at the correct class, and INLINEFORM15 is the predicted probability distribution from the softmax layer  The training objective then takes the following form: DISPLAYFORM0
where INLINEFORM0 is the calculated representative vector of the audio signal with dimensionality INLINEFORM1   The INLINEFORM2 and the bias INLINEFORM3 are learned model parameters  C is the total number of classes, and N is the total number of samples used in training  The upper part of Figure shows the architecture of the ARE model 
Text Recurrent Encoder (TRE)
We assume that speech transcripts can be extracted from audio signals with high accuracy, given the advancement of ASR technologies BIBREF7   We attempt to use the processed textual information as another modality in predicting the emotion class of a given signal  To use textual information, a speech transcript is tokenized and indexed into a sequence of tokens using the Natural Language Toolkit (NLTK) BIBREF27   Each token is then passed through a word-embedding layer that converts a word index to a corresponding 300-dimensional vector that contains additional contextual meaning between words  The sequence of embedded tokens is fed into a text recurrent encoder (TRE) in such a way that the audio MFCC features are encoded using the ARE represented by equation EQREF2   In this case, INLINEFORM0 is the t- INLINEFORM1 embedded token from the text input  Finally, the emotion class is predicted from the last hidden state of the text-RNN using the softmax function 
We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 is last hidden state of the text-RNN, INLINEFORM1 , and the INLINEFORM2 and bias INLINEFORM3 are learned model parameters  The lower part of Figure indicates the architecture of the TRE model 
Multimodal Dual Recurrent Encoder (MDRE)
We present a novel architecture called the multimodal dual recurrent encoder (MDRE) to overcome the limitations of existing approaches  In this study, we consider multiple modalities, such as MFCC features, prosodic features and transcripts, which contain sequential audio information, statistical audio information and textual information, respectively  These types of data are the same as those used in the ARE and TRE cases  The MDRE model employs two RNNs to encode data from the audio signal and textual inputs independently  The audio-RNN encodes MFCC features from the audio signal using equation EQREF2   The last hidden state of the audio-RNN is concatenated with the prosodic features to form the final vector representation INLINEFORM0 , and this vector is then passed through a fully connected neural network layer to form the audio encoding vector A  On the other hand, the text-RNN encodes the word sequence of the transcript using equation EQREF2   The final hidden states of the text-RNN are also passed through another fully connected neural network layer to form a textual encoding vector T  Finally, the emotion class is predicted by applying the softmax function to the concatenation of the vectors A and T  We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 is the feed-forward neural network with weight parameter INLINEFORM1 , and INLINEFORM2 , INLINEFORM3 are final encoding vectors from the audio-RNN and text-RNN, respectively  INLINEFORM4 and the bias INLINEFORM5 are learned model parameters 
Multimodal Dual Recurrent Encoder with Attention (MDREA)
Inspired by the concept of the attention mechanism used in neural machine translation BIBREF28 , we propose a novel multimodal attention method to focus on the specific parts of a transcript that contain strong emotional information, conditioning on the audio information  Figure shows the architecture of the MDREA model  First, the audio data and text data are encoded with the audio-RNN and text-RNN using equation EQREF2   We then consider the final audio encoding vector INLINEFORM0 as a context vector  As seen in equation EQREF9 , during each time step t, the dot product between the context vector e and the hidden state of the text-RNN at each t-th sequence INLINEFORM1 is evaluated to calculate a similarity score INLINEFORM2   Using this score INLINEFORM3 as a weight parameter, the weighted sum of the sequences of the hidden state of the text-RNN, INLINEFORM4 , is calculated to generate an attention-application vector Z  This attention-application vector is concatenated with the final encoding vector of the audio-RNN INLINEFORM5 (equation EQREF7 ), which will be passed through the softmax function to predict the emotion class  We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 and the bias INLINEFORM1 are learned model parameters 
Dataset
We evaluate our model using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) BIBREF18 dataset  This dataset was collected following theatrical theory in order to simulate natural dyadic interactions between actors  We use categorical evaluations with majority agreement  We use only four emotional categories happy, sad, angry, and neutral to compare the performance of our model with other research using the same categories  The IEMOCAP dataset includes five sessions, and each session contains utterances from two speakers (one male and one female)  This data collection process resulted in 10 unique speakers  For consistent comparison with previous work, we merge the excitement dataset with the happiness dataset  The final dataset contains a total of 5531 utterances (1636 happy, 1084 sad, 1103 angry, 1708 neutral) 
Feature extraction
To extract speech information from audio signals, we use MFCC values, which are widely used in analyzing audio signals  The MFCC feature set contains a total of 39 features, which include 12 MFCC parameters (1-12) from the 26 Mel-frequency bands and log-energy parameters, 13 delta and 13 acceleration coefficients The frame size is set to 25 ms at a rate of 10 ms with the Hamming function  According to the length of each wave file, the sequential step of the MFCC features is varied  To extract additional information from the data, we also use prosodic features, which show effectiveness in affective computing  The prosodic features are composed of 35 features, which include the F0 frequency, the voicing probability, and the loudness contours  All of these MFCC and prosodic features are extracted from the data using the OpenSMILE toolkit BIBREF26  
Implementation details
Among the variants of the RNN function, we use GRUs as they yield comparable performance to that of the LSTM and include a smaller number of weight parameters BIBREF29   We use a max encoder step of 750 for the audio input, based on the implementation choices presented in BIBREF30 and 128 for the text input because it covers the maximum length of the transcripts  The vocabulary size of the dataset is 3,747, including the “_UNK_" token, which represents unknown words, and the “_PAD_" token, which is used to indicate padding information added while preparing mini-batch data  The number of hidden units and the number of layers in the RNN for each model (ARE, TRE, MDRE and MDREA) are selected based on extensive hyperparameter search experiments  The weights of the hidden units are initialized using orthogonal weights BIBREF31 ], and the text embedding layer is initialized from pretrained word-embedding vectors BIBREF32  
In preparing the textual dataset, we first use the released transcripts of the IEMOCAP dataset for simplicity  To investigate the practical performance, we then process all of the IEMOCAP audio data using an ASR system (the Google Cloud Speech API) and retrieve the transcripts  The performance of the Google ASR system is reflected by its word error rate (WER) of 5 53% 
Performance evaluation
As the dataset is not explicitly split beforehand into training, development, and testing sets, we perform 5-fold cross validation to determine the overall performance of the model  The data in each fold are split into training, development, and testing datasets (8:0 5:1 5, respectively)  After training the model, we measure the weighted average precision (WAP) over the 5-fold dataset  We train and evaluate the model 10 times per fold, and the model performance is assessed in terms of the mean score and standard deviation 
We examine the WAP values, which are shown in Table 1  First, our ARE model shows the baseline performance because we use minimal audio features, such as the MFCC and prosodic features with simple architectures  On the other hand, the TRE model shows higher performance gain compared to the ARE  From this result, we note that textual data are informative in emotion prediction tasks, and the recurrent encoder model is effective in understanding these types of sequential data  Second, the newly proposed model, MDRE, shows a substantial performance gain  It thus achieves the state-of-the-art performance with a WAP value of 0 718  This result shows that multimodal information is a key factor in affective computing  Lastly, the attention model, MDREA, also outperforms the best existing research results (WAP 0 690 to 0 688) BIBREF19   However, the MDREA model does not match the performance of the MDRE model, even though it utilizes a more complex architecture  We believe that this result arises because insufficient data are available to properly determine the complex model parameters in the MDREA model  Moreover, we presume that this model will show better performance when the audio signals are aligned with the textual sequence while applying the attention mechanism  We leave the implementation of this point as a future research direction 
To investigate the practical performance of the proposed models, we conduct further experiments with the ASR-processed transcript data (see “-ASR” models in Table )  The label accuracy of the processed transcripts is 5 53% WER  The TRE-ASR, MDRE-ASR and MDREA-ASR models reflect degraded performance compared to that of the TRE, MDRE and MDREA models  However, the performance of these models is still competitive; in particular, the MDRE-ASR model outperforms the previous best-performing model, 3CNN-LSTM10H (WAP 0 691 to 0 688) 
Error analysis
We analyze the predictions of the ARE, TRE, and MDRE models  Figure shows the confusion matrix of each model  The ARE model (Fig  ) incorrectly classifies most instances of happy as neutral (43 51%); thus, it shows reduced accuracy (35 15%) in predicting the the happy class  Overall, most of the emotion classes are frequently confused with the neutral class  This observation is in line with the findings of BIBREF30 , who noted that the neutral class is located in the center of the activation-valence space, complicating its discrimination from the other classes 
Interestingly, the TRE model (Fig  ) shows greater prediction gains in predicting the happy class when compared to the ARE model (35 15% to 75 73%)  This result seems plausible because the model can benefit from the differences among the distributions of words in happy and neutral expressions, which gives more emotional information to the model than that of the audio signal data  On the other hand, it is striking that the TRE model incorrectly predicts instances of the sad class as the happy class 16 20% of the time, even though these emotional states are opposites of one another 
The MDRE model (Fig  ) compensates for the weaknesses of the previous two models (ARE and TRE) and benefits from their strengths to a surprising degree  The values arranged along the diagonal axis show that all of the accuracies of the correctly predicted class have increased  Furthermore, the occurrence of the incorrect “sad-to-happy" cases in the TRE model is reduced from 16 20% to 9 15% 
Conclusions
In this paper, we propose a novel multimodal dual recurrent encoder model that simultaneously utilizes text data, as well as audio signals, to permit the better understanding of speech data  Our model encodes the information from audio and text sequences using dual RNNs and then combines the information from these sources using a feed-forward neural model to predict the emotion class  Extensive experiments show that our proposed model outperforms other state-of-the-art methods in classifying the four emotion categories, and accuracies ranging from 68 8% to 71 8% are obtained when the model is applied to the IEMOCAP dataset  In particular, it resolves the issue in which predictions frequently incorrectly yield the neutral class, as occurs in previous models that focus on audio features 
In the future work, we aim to extend the modalities to audio, text and video inputs  Furthermore, we plan to investigate the application of the attention mechanism to data derived from multiple modalities  This approach seems likely to uncover enhanced learning schemes that will increase performance in both speech emotion recognition and other multimodal classification tasks 
Acknowledgments
K  Jung is with the Department of Electrical and Computer Engineering, ASRI, Seoul National University, Seoul, Korea  This work was supported by the Ministry of Trade, Industry & Energy (MOTIE, Korea) under Industrial Technology Innovation Program (No 10073144) 
Context 3: Introduction
Offensive content has become pervasive in social media and a reason of concern for government organizations, online communities, and social media platforms  One of the most common strategies to tackle the problem is to train systems capable of recognizing offensive content, which then can be deleted or set aside for human moderation  In the last few years, there have been several studies published on the application of computational methods to deal with this problem  Most prior work focuses on a different aspect of offensive language such as abusive language BIBREF0 , BIBREF1 , (cyber-)aggression BIBREF2 , (cyber-)bullying BIBREF3 , BIBREF4 , toxic comments INLINEFORM0 , hate speech BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , and offensive language BIBREF11   Prior work has focused on these aspects of offensive language in Twitter BIBREF3 , BIBREF7 , BIBREF8 , BIBREF11 , Wikipedia comments, and Facebook posts BIBREF2  
Recently, Waseem et  al  ( BIBREF12 ) acknowledged the similarities among prior work and discussed the need for a typology that differentiates between whether the (abusive) language is directed towards a specific individual or entity or towards a generalized group and whether the abusive content is explicit or implicit  Wiegand et al  ( BIBREF11 ) followed this trend as well on German tweets  In their evaluation, they have a task to detect offensive vs not offensive tweets and a second task for distinguishing between the offensive tweets as profanity, insult, or abuse  However, no prior work has explored the target of the offensive language, which is important in many scenarios, e g , when studying hate speech with respect to a specific target 
Therefore, we expand on these ideas by proposing a a hierarchical three-level annotation model that encompasses:
Using this annotation model, we create a new large publicly available dataset of English tweets  The key contributions of this paper are as follows:
Related Work
Different abusive and offense language identification sub-tasks have been explored in the past few years including aggression identification, bullying detection, hate speech, toxic comments, and offensive language 
Aggression identification: The TRAC shared task on Aggression Identification BIBREF2 provided participants with a dataset containing 15,000 annotated Facebook posts and comments in English and Hindi for training and validation  For testing, two different sets, one from Facebook and one from Twitter were provided  Systems were trained to discriminate between three classes: non-aggressive, covertly aggressive, and overtly aggressive 
Bullying detection: Several studies have been published on bullying detection  One of them is the one by xu2012learning which apply sentiment analysis to detect bullying in tweets  xu2012learning use topic models to to identify relevant topics in bullying  Another related study is the one by dadvar2013improving which use user-related features such as the frequency of profanity in previous messages to improve bullying detection 
Hate speech identification: It is perhaps the most widespread abusive language detection sub-task  There have been several studies published on this sub-task such as kwok2013locate and djuric2015hate who build a binary classifier to distinguish between `clean' comments and comments containing hate speech and profanity  More recently, Davidson et al  davidson2017automated presented the hate speech detection dataset containing over 24,000 English tweets labeled as non offensive, hate speech, and profanity 
Offensive language: The GermEval BIBREF11 shared task focused on Offensive language identification in German tweets  A dataset of over 8,500 annotated tweets was provided for a course-grained binary classification task in which systems were trained to discriminate between offensive and non-offensive tweets and a second task where the organizers broke down the offensive class into three classes: profanity, insult, and abuse 
Toxic comments: The Toxic Comment Classification Challenge was an open competition at Kaggle which provided participants with comments from Wikipedia labeled in six classes: toxic, severe toxic, obscene, threat, insult, identity hate 
While each of these sub-tasks tackle a particular type of abuse or offense, they share similar properties and the hierarchical annotation model proposed in this paper aims to capture this  Considering that, for example, an insult targeted at an individual is commonly known as cyberbulling and that insults targeted at a group are known as hate speech, we pose that OLID's hierarchical annotation model makes it a useful resource for various offensive language identification sub-tasks 
Hierarchically Modelling Offensive Content
In the OLID dataset, we use a hierarchical annotation model split into three levels to distinguish between whether language is offensive or not (A), and type (B) and target (C) of the offensive language  Each level is described in more detail in the following subsections and examples are shown in Table TABREF10  
Level A: Offensive language Detection
Level A discriminates between offensive (OFF) and non-offensive (NOT) tweets 
Not Offensive (NOT): Posts that do not contain offense or profanity;
Offensive (OFF): We label a post as offensive if it contains any form of non-acceptable language (profanity) or a targeted offense, which can be veiled or direct  This category includes insults, threats, and posts containing profane language or swear words 
Level B: Categorization of Offensive Language
Level B categorizes the type of offense and two labels are used: targeted (TIN) and untargeted (INT) insults and threats 
Targeted Insult (TIN): Posts which contain an insult/threat to an individual, group, or others (see next layer);
Untargeted (UNT): Posts containing non-targeted profanity and swearing  Posts with general profanity are not targeted, but they contain non-acceptable language 
Level C: Offensive Language Target Identification
Level C categorizes the targets of insults and threats as individual (IND), group (GRP), and other (OTH) 
Individual (IND): Posts targeting an individual  It can be a a famous person, a named individual or an unnamed participant in the conversation  Insults and threats targeted at individuals are often defined as cyberbulling 
Group (GRP): The target of these offensive posts is a group of people considered as a unity due to the same ethnicity, gender or sexual orientation, political affiliation, religious belief, or other common characteristic  Many of the insults and threats targeted at a group correspond to what is commonly understood as hate speech 
Other (OTH): The target of these offensive posts does not belong to any of the previous two categories (e g  an organization, a situation, an event, or an issue) 
Data Collection
The data included in OLID has been collected from Twitter  We retrieved the data using the Twitter API by searching for keywords and constructions that are often included in offensive messages, such as `she is' or `to:BreitBartNews'  We carried out a first round of trial annotation of 300 instances with six experts  The goal of the trial annotation was to 1) evaluate the proposed tagset; 2) evaluate the data retrieval method; and 3) create a gold standard with instances that could be used as test questions in the training and test setting annotation which was carried out using crowdsourcing  The breakdown of keywords and their offensive content in the trial data of 300 tweets is shown in Table TABREF14   We included a left (@NewYorker) and far-right (@BreitBartNews) news accounts because there tends to be political offense in the comments  One of the best offensive keywords was tweets that were flagged as not being safe by the Twitter `safe' filter (the `-' indicates `not safe')  The vast majority of content on Twitter is not offensive so we tried different strategies to keep a reasonable number of tweets in the offensive class amounting to around 30% of the dataset including excluding some keywords that were not high in offensive content such as `they are` and `to:NewYorker`  Although `he is' is lower in offensive content we kept it as a keyword to avoid gender bias  In addition to the keywords in the trial set, we searched for more political keywords which tend to be higher in offensive content, and sampled our dataset such that 50% of the the tweets come from political keywords and 50% come from non-political keywords  In addition to the keywords `gun control', and `to:BreitbartNews', political keywords used to collect these tweets are `MAGA', `antifa', `conservative' and `liberal'  We computed Fliess' INLINEFORM0 on the trial set for the five annotators on 21 of the tweets  INLINEFORM1 is  83 for Layer A (OFF vs NOT) indicating high agreement  As to normalization and anonymization, no user metadata or Twitter IDs have been stored, and URLs and Twitter mentions have been substituted to placeholders  We follow prior work in related areas (burnap2015cyber,davidson2017automated) and annotate our data using crowdsourcing using the platform Figure Eight  We ensure data quality by: 1) we only received annotations from individuals who were experienced in the platform; and 2) we used test questions to discard annotations of individuals who did not reach a certain threshold  Each instance in the dataset was annotated by multiple annotators and inter-annotator agreement has been calculated  We first acquired two annotations for each instance  In case of 100% agreement, we considered these as acceptable annotations, and in case of disagreement, we requested more annotations until the agreement was above 66%  After the crowdsourcing annotation, we used expert adjudication to guarantee the quality of the annotation  The breakdown of the data into training and testing for the labels from each level is shown in Table TABREF15  
Experiments and Evaluation
We assess our dataset using traditional and deep learning methods  Our simplest model is a linear SVM trained on word unigrams  SVMs have produced state-of-the-art results for many text classification tasks BIBREF13   We also train a bidirectional Long Short-Term-Memory (BiLSTM) model, which we adapted from the sentiment analysis system of sentimentSystem,rasooli2018cross and altered to predict offensive labels instead  It consists of (1) an input embedding layer, (2) a bidirectional LSTM layer, (3) an average pooling layer of input features  The concatenation of the LSTM's and average pool layer is passed through a dense layer and the output is passed through a softmax function  We set two input channels for the input embedding layers: pre-trained FastText embeddings BIBREF14 , as well as updatable embeddings learned by the model during training  Finally, we also apply a Convolutional Neural Network (CNN) model based on the architecture of BIBREF15 , using the same multi-channel inputs as the above BiLSTM 
Our models are trained on the training data, and evaluated by predicting the labels for the held-out test set  The distribution is described in Table TABREF15   We evaluate and compare the models using the macro-averaged F1-score as the label distribution is highly imbalanced  Per-class Precision (P), Recall (R), and F1-score (F1), also with other averaged metrics are also reported  The models are compared against baselines of predicting all labels as the majority or minority classes 
Offensive Language Detection
The performance on discriminating between offensive (OFF) and non-offensive (NOT) posts is reported in Table TABREF18   We can see that all systems perform significantly better than chance, with the neural models being substantially better than the SVM  The CNN outperforms the RNN model, achieving a macro-F1 score of 0 80 
Categorization of Offensive Language
In this experiment, the two systems were trained to discriminate between insults and threats (TIN) and untargeted (UNT) offenses, which generally refer to profanity  The results are shown in Table TABREF19  
The CNN system achieved higher performance in this experiment compared to the BiLSTM, with a macro-F1 score of 0 69  All systems performed better at identifying target and threats (TIN) than untargeted offenses (UNT) 
Offensive Language Target Identification
The results of the offensive target identification experiment are reported in Table TABREF20   Here the systems were trained to distinguish between three targets: a group (GRP), an individual (IND), or others (OTH)  All three models achieved similar results far surpassing the random baselines, with a slight performance edge for the neural models 
The performance of all systems for the OTH class is 0  This poor performances can be explained by two main factors  First, unlike the two other classes, OTH is a heterogeneous collection of targets  It includes offensive tweets targeted at organizations, situations, events, etc  making it more challenging for systems to learn discriminative properties of this class  Second, this class contains fewer training instances than the other two  There are only 395 instances in OTH, and 1,075 in GRP, and 2,407 in IND 
Conclusion and Future Work
This paper presents OLID, a new dataset with annotation of type and target of offensive language  OLID is the official dataset of the shared task SemEval 2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval) BIBREF16   In OffensEval, each annotation level in OLID is an independent sub-task  The dataset contains 14,100 tweets and is released freely to the research community  To the best of our knowledge, this is the first dataset to contain annotation of type and target of offenses in social media and it opens several new avenues for research in this area  We present baseline experiments using SVMs and neural networks to identify the offensive tweets, discriminate between insults, threats, and profanity, and finally to identify the target of the offensive messages  The results show that this is a challenging task  A CNN-based sentence classifier achieved the best results in all three sub-tasks 
In future work, we would like to make a cross-corpus comparison of OLID and datasets annotated for similar tasks such as aggression identification BIBREF2 and hate speech detection BIBREF8   This comparison is, however, far from trivial as the annotation of OLID is different 
Acknowledgments
The research presented in this paper was partially supported by an ERAS fellowship awarded by the University of Wolverhampton 

ground_truth: Metric difference between Aloha and best baseline score:
Hits@1/20: +0.061 (0.3642 vs 0.3032)
MRR: +0.0572(0.5114 vs 0.4542)
F1: -0.0484 (0.3901 vs 0.4385)
BLEU: +0.0474 (0.2867 vs 0.2393)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. By how much does their model outperform the state of the art results?
Context 4: Introduction
Data annotation is a major bottleneck for the application of supervised learning approaches to many problems  As a result, unsupervised methods that learn directly from unlabeled data are increasingly important  For tasks related to unsupervised syntactic analysis, discrete generative models have dominated in recent years – for example, for both part-of-speech (POS) induction BIBREF0 , BIBREF1 and unsupervised dependency parsing BIBREF2 , BIBREF3 , BIBREF4   While similar models have had success on a range of unsupervised tasks, they have mostly ignored the apparent utility of continuous word representations evident from supervised NLP applications BIBREF5 , BIBREF6   In this work, we focus on leveraging and explicitly representing continuous word embeddings within unsupervised models of syntactic structure 
Pre-trained word embeddings from massive unlabeled corpora offer a compact way of injecting a prior notion of word similarity into models that would otherwise treat words as discrete, isolated categories  However, the specific properties of language captured by any particular embedding scheme can be difficult to control, and, further, may not be ideally suited to the task at hand  For example, pre-trained skip-gram embeddings BIBREF7 with small context window size are found to capture the syntactic properties of language well BIBREF8 , BIBREF9   However, if our goal is to separate syntactic categories, this embedding space is not ideal – POS categories correspond to overlapping interspersed regions in the embedding space, evident in Figure SECREF4  
In our approach, we propose to learn a new latent embedding space as a projection of pre-trained embeddings (depicted in Figure SECREF5 ), while jointly learning latent syntactic structure – for example, POS categories or syntactic dependencies  To this end, we introduce a new generative model (shown in Figure FIGREF6 ) that first generates a latent syntactic representation (e g  a dependency parse) from a discrete structured prior (which we also call the “syntax model”), then, conditioned on this representation, generates a sequence of latent embedding random variables corresponding to each word, and finally produces the observed (pre-trained) word embeddings by projecting these latent vectors through a parameterized non-linear function  The latent embeddings can be jointly learned with the structured syntax model in a completely unsupervised fashion 
By choosing an invertible neural network as our non-linear projector, and then parameterizing our model in terms of the projection's inverse, we are able to derive tractable exact inference and marginal likelihood computation procedures so long as inference is tractable in the underlying syntax model  In sec:learn-with-inv we show that this derivation corresponds to an alternate view of our approach whereby we jointly learn a mapping of observed word embeddings to a new embedding space that is more suitable for the syntax model, but include an additional Jacobian regularization term to prevent information loss 
Recent work has sought to take advantage of word embeddings in unsupervised generative models with alternate approaches BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12   BIBREF9 build an HMM with Gaussian emissions on observed word embeddings, but they do not attempt to learn new embeddings  BIBREF10 , BIBREF11 , and BIBREF12 extend HMM or dependency model with valence (DMV) BIBREF2 with multinomials that use word (or tag) embeddings in their parameterization  However, they do not represent the embeddings as latent variables 
In experiments, we instantiate our approach using both a Markov-structured syntax model and a tree-structured syntax model – specifically, the DMV  We evaluate on two tasks: part-of-speech (POS) induction and unsupervised dependency parsing without gold POS tags  Experimental results on the Penn Treebank BIBREF13 demonstrate that our approach improves the basic HMM and DMV by a large margin, leading to the state-of-the-art results on POS induction, and state-of-the-art results on unsupervised dependency parsing in the difficult training scenario where neither gold POS annotation nor punctuation-based constraints are available 
Model
As an illustrative example, we first present a baseline model for Markov syntactic structure (POS induction) that treats a sequence of pre-trained word embeddings as observations  Then, we propose our novel approach, again using Markov structure, that introduces latent word embedding variables and a neural projector  Lastly, we extend our approach to more general syntactic structures 
Example: Gaussian HMM
We start by describing the Gaussian hidden Markov model introduced by BIBREF9 , which is a locally normalized model with multinomial transitions and Gaussian emissions  Given a sentence of length INLINEFORM0 , we denote the latent POS tags as INLINEFORM1 , observed (pre-trained) word embeddings as INLINEFORM2 , transition parameters as INLINEFORM3 , and Gaussian emission parameters as INLINEFORM4   The joint distribution of data and latent variables factors as:
DISPLAYFORM0
where INLINEFORM0 is the multinomial transition probability and INLINEFORM1 is the multivariate Gaussian emission probability 
While the observed word embeddings do inform this model with a notion of word similarity – lacking in the basic multinomial HMM – the Gaussian emissions may not be sufficiently flexible to separate some syntactic categories in the complex pre-trained embedding space – for example the skip-gram embedding space as visualized in Figure SECREF4 where different POS categories overlap  Next we introduce a new approach that adds flexibility to the emission distribution by incorporating new latent embedding variables 
Markov Structure with Neural Projector
To flexibly model observed embeddings and yield a new representation space that is more suitable for the syntax model, we propose to cascade a neural network as a projection function, deterministically transforming the simple space defined by the Gaussian HMM to the observed embedding space  We denote the latent embedding of the INLINEFORM0 word in a sentence as INLINEFORM1 , and the neural projection function as INLINEFORM2 , parameterized by INLINEFORM3   In the case of sequential Markov structure, our new model corresponds to the following generative process:
For each time step INLINEFORM0 ,
[noitemsep, leftmargin=*]
Draw the latent state INLINEFORM0
Draw the latent embedding INLINEFORM0
Deterministically produce embedding
INLINEFORM0
The graphical model is depicted in Figure FIGREF6   The deterministic projection can also be viewed as sampling each observation from a point mass at INLINEFORM0   The joint distribution of our model is: DISPLAYFORM0
where INLINEFORM0 is a conditional Gaussian distribution, and INLINEFORM1 is the Dirac delta function centered at INLINEFORM2 : DISPLAYFORM0
General Structure with Neural Projector
Our approach can be applied to a broad family of structured syntax models  We denote latent embedding variables as INLINEFORM0 , discrete latent variables in the syntax model as INLINEFORM1 ( INLINEFORM2 ), where INLINEFORM3 are conditioned to generate INLINEFORM4   The joint probability of our model factors as:
DISPLAYFORM0
where INLINEFORM0 represents the probability of the syntax model, and can encode any syntactic structure – though, its factorization structure will determine whether inference is tractable in our full model  As shown in Figure FIGREF6 , we focus on two syntax models for syntactic analysis in this paper  The first is Markov-structured, which we use for POS induction, and the second is DMV-structured, which we use to learn dependency parses without supervision 
The marginal data likelihood of our model is: DISPLAYFORM0
While the discrete variables INLINEFORM0 can be marginalized out with dynamic program in many cases, it is generally intractable to marginalize out the latent continuous variables, INLINEFORM1 , for an arbitrary projection INLINEFORM2 in Eq  ( EQREF17 ), which means inference and learning may be difficult  In sec:opt, we address this issue by constraining INLINEFORM3 to be invertible, and show that this constraint enables tractable exact inference and marginal likelihood computation 
Learning & Inference
In this section, we introduce an invertibility condition for our neural projector to tackle the optimization challenge  Specifically, we constrain our neural projector with two requirements: (1) INLINEFORM0 and (2) INLINEFORM1 exists  Invertible transformations have been explored before in independent components analysis BIBREF14 , gaussianization BIBREF15 , and deep density models BIBREF16 , BIBREF17 , BIBREF18 , for unstructured data  Here, we generalize this style of approach to structured learning, and augment it with discrete latent variables ( INLINEFORM2 )  Under the invertibility condition, we derive a learning algorithm and give another view of our approach revealed by the objective function  Then, we present the architecture of a neural projector we use in experiments: a volume-preserving invertible neural network proposed by BIBREF16 for independent components estimation 
Learning with Invertibility
For ease of exposition, we explain the learning algorithm in terms of Markov structure without loss of generality  As shown in Eq  ( EQREF17 ), the optimization challenge in our approach comes from the intractability of the marginalized emission factor INLINEFORM0   If we can marginalize out INLINEFORM1 and compute INLINEFORM2 , then the posterior and marginal likelihood of our Markov-structured model can be computed with the forward-backward algorithm  We can apply Eq  ( EQREF14 ) and obtain : INLINEFORM3
By using the change of variable rule to the integration, which allows the integration variable INLINEFORM0 to be replaced by INLINEFORM1 , the marginal emission factor can be computed in closed-form when the invertibility condition is satisfied: DISPLAYFORM0
where INLINEFORM0 is a conditional Gaussian distribution, INLINEFORM1 is the Jacobian matrix of function INLINEFORM2 at INLINEFORM3 , and INLINEFORM4 represents the absolute value of its determinant  This Jacobian term is nonzero and differentiable if and only if INLINEFORM5 exists 
Eq  ( EQREF19 ) shows that we can directly calculate the marginal emission distribution INLINEFORM0   Denote the marginal data likelihood of Gaussian HMM as INLINEFORM1 , then the log marginal data likelihood of our model can be directly written as: DISPLAYFORM0
where INLINEFORM0 represents the new sequence of embeddings after applying INLINEFORM1 to each INLINEFORM2   Eq  ( EQREF20 ) shows that the training objective of our model is simply the Gaussian HMM log likelihood with an additional Jacobian regularization term  From this view, our approach can be seen as equivalent to reversely projecting the data through INLINEFORM3 to another manifold INLINEFORM4 that is directly modeled by the Gaussian HMM, with a regularization term  Intuitively, we optimize the reverse projection INLINEFORM5 to modify the INLINEFORM6 space, making it more appropriate for the syntax model  The Jacobian regularization term accounts for the volume expansion or contraction behavior of the projection  Maximizing it can be thought of as preventing information loss  In the extreme case, the Jacobian determinant is equal to zero, which means the projection is non-invertible and thus information is being lost through the projection  Such “information preserving” regularization is crucial during optimization, otherwise the trivial solution of always projecting data to the same single point to maximize likelihood is viable 
More generally, for an arbitrary syntax model the data likelihood of our approach is: DISPLAYFORM0
If the syntax model itself allows for tractable inference and marginal likelihood computation, the same dynamic program can be used to marginalize out INLINEFORM0   Therefore, our joint model inherits the tractability of the underlying syntax model 
Invertible Volume-Preserving Neural Net
For the projection we can use an arbitrary invertible function, and given the representational power of neural networks they seem a natural choice  However, calculating the inverse and Jacobian of an arbitrary neural network can be difficult, as it requires that all component functions be invertible and also requires storage of large Jacobian matrices, which is memory intensive  To address this issue, several recent papers propose specially designed invertible networks that are easily trainable yet still powerful BIBREF16 , BIBREF17 , BIBREF19   Inspired by these works, we use the invertible transformation proposed by BIBREF16 , which consists of a series of “coupling layers”  This architecture is specially designed to guarantee a unit Jacobian determinant (and thus the invertibility property) 
From Eq  ( EQREF22 ) we know that only INLINEFORM0 is required for accomplishing learning and inference; we never need to explicitly construct INLINEFORM1   Thus, we directly define the architecture of INLINEFORM2   As shown in Figure FIGREF24 , the nonlinear transformation from the observed embedding INLINEFORM3 to INLINEFORM4 represents the first coupling layer  The input in this layer is partitioned into left and right halves of dimensions, INLINEFORM5 and INLINEFORM6 , respectively  A single coupling layer is defined as: DISPLAYFORM0
where INLINEFORM0 is the coupling function and can be any nonlinear form  This transformation satisfies INLINEFORM1 , and BIBREF16 show that its Jacobian matrix is triangular with all ones on the main diagonal  Thus the Jacobian determinant is always equal to one (i e  volume-preserving) and the invertibility condition is naturally satisfied 
To be sufficiently expressive, we compose multiple coupling layers as suggested in BIBREF16   Specifically, we exchange the role of left and right half vectors at each layer as shown in Figure FIGREF24   For instance, from INLINEFORM0 to INLINEFORM1 the left subset INLINEFORM2 is unchanged, while from INLINEFORM3 to INLINEFORM4 the right subset INLINEFORM5 remains the same  Also note that composing multiple coupling layers does not change the volume-preserving and invertibility properties  Such a sequence of invertible transformations from the data space INLINEFORM6 to INLINEFORM7 is also called normalizing flow BIBREF20  
Experiments
In this section, we first describe our datasets and experimental setup  We then instantiate our approach with Markov and DMV-structured syntax models, and report results on POS tagging and dependency grammar induction respectively  Lastly, we analyze the learned latent embeddings 
Data
For both POS tagging and dependency parsing, we run experiments on the Wall Street Journal (WSJ) portion of the Penn Treebank  To create the observed data embeddings, we train skip-gram word embeddings BIBREF7 that are found to capture syntactic properties well when trained with small context window BIBREF8 , BIBREF9   Following BIBREF9 , the dimensionality INLINEFORM0 is set to 100, and the training context window size is set to 1 to encode more syntactic information  The skip-gram embeddings are trained on the one billion word language modeling benchmark dataset BIBREF21 in addition to the WSJ corpus 
General Experimental Setup
For the neural projector, we employ rectified networks as coupling function INLINEFORM0 following BIBREF16   We use a rectified network with an input layer, one hidden layer, and linear output units, the number of hidden units is set to the same as the number of input units  The number of coupling layers are varied as 4, 8, 16 for both tasks

ground_truth: the attention model, MDREA, also outperforms the best existing research results (WAP 0.690 to 0.688)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. In what language are the tweets?
Context 4:  We optimize marginal data likelihood directly using Adam BIBREF22   For both tasks in the fully unsupervised setting, we do not tune the hyper-parameters using supervised data 
Unsupervised POS tagging
For unsupervised POS tagging, we use a Markov-structured syntax model in our approach, which is a popular structure for unsupervised tagging tasks BIBREF9 , BIBREF10  
Following existing literature, we train and test on the entire WSJ corpus (49208 sentences, 1M tokens)  We use 45 tag clusters, the number of POS tags that appear in WSJ corpus  We train the discrete HMM and the Gaussian HMM BIBREF9 as baselines  For the Gaussian HMM, mean vectors of Gaussian emissions are initialized with the empirical mean of all word vectors with an additive noise  We assume diagonal covariance matrix for INLINEFORM0 and initialize it with the empirical variance of the word vectors  Following BIBREF9 , the covariance matrix is fixed during training  The multinomial probabilities are initialized as INLINEFORM1 , where INLINEFORM2   For our approach, we initialize the syntax model and Gaussian parameters with the pre-trained Gaussian HMM  The weights of layers in the rectified network are initialized from a uniform distribution with mean zero and a standard deviation of INLINEFORM3 , where INLINEFORM4 is the input dimension  We evaluate the performance of POS tagging with both Many-to-One (M-1) accuracy BIBREF23 and V-Measure (VM) BIBREF24   Given a model we found that the tagging performance is well-correlated with the training data likelihood, thus we use training data likelihood as a unsupervised criterion to select the trained model over 10 random restarts after training 50 epochs  We repeat this process 5 times and report the mean and standard deviation of performance 
We compare our approach with basic HMM, Gaussian HMM, and several state-of-the-art systems, including sophisticated HMM variants and clustering techniques with hand-engineered features  The results are presented in Table TABREF32   Through the introduced latent embeddings and additional neural projection, our approach improves over the Gaussian HMM by 5 4 points in M-1 and 5 6 points in VM  Neural HMM (NHMM) BIBREF10 is a baseline that also learns word representation jointly  Both their basic model and extended Conv version does not outperform the Gaussian HMM  Their best model incorporates another LSTM to model long distance dependency and breaks the Markov assumption, yet our approach still achieves substantial improvement over it without considering more context information  Moreover, our method outperforms the best published result that benefits from hand-engineered features BIBREF27 by 2 0 points on VM 
We found that most tagging errors happen in noun subcategories  Therefore, we do the one-to-one mapping between gold POS tags and induced clusters and plot the normalized confusion matrix of noun subcategories in Figure FIGREF35   The Gaussian HMM fails to identify “NN” and “NNS” correctly for most cases, and it often recognizes “NNPS” as “NNP”  In contrast, our approach corrects these errors well 
Unsupervised Dependency Parsing without gold POS tags
For the task of unsupervised dependency parse induction, we employ the Dependency Model with Valence (DMV) BIBREF2 as the syntax model in our approach  DMV is a generative model that defines a probability distribution over dependency parse trees and syntactic categories, generating tokens and dependencies in a head-outward fashion  While, traditionally, DMV is trained using gold POS tags as observed syntactic categories, in our approach, we treat each tag as a latent variable, as described in sec:general-neural 
Most existing approaches to this task are not fully unsupervised since they rely on gold POS tags following the original experimental setup for DMV  This is partially because automatically parsing from words is difficult even when using unsupervised syntactic categories BIBREF29   However, inducing dependencies from words alone represents a more realistic experimental condition since gold POS tags are often unavailable in practice  Previous work that has trained from words alone often requires additional linguistic constraints (like sentence internal boundaries) BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , acoustic cues BIBREF33 , additional training data BIBREF4 , or annotated data from related languages BIBREF34   Our approach is naturally designed to train on word embeddings directly, thus we attempt to induce dependencies without using gold POS tags or other extra linguistic information 
Like previous work we use sections 02-21 of WSJ corpus as training data and evaluate on section 23, we remove punctuations and train the models on sentences of length INLINEFORM0 , “head-percolation” rules BIBREF39 are applied to obtain gold dependencies for evaluation  We train basic DMV, extended DMV (E-DMV) BIBREF35 and Gaussian DMV (which treats POS tag as unknown latent variables and generates observed word embeddings directly conditioned on them following Gaussian distribution) as baselines  Basic DMV and E-DMV are trained with Viterbi EM BIBREF40 on unsupervised POS tags induced from our Markov-structured model described in sec:pos  Multinomial parameters of the syntax model in both Gaussian DMV and our model are initialized with the pre-trained DMV baseline  Other parameters are initialized in the same way as in the POS tagging experiment  The directed dependency accuracy (DDA) is used for evaluation and we report accuracy on sentences of length INLINEFORM1 and all lengths  We train the parser until training data likelihood converges, and report the mean and standard deviation over 20 random restarts 
Our model directly observes word embeddings and does not require gold POS tags during training  Thus, results from related work trained on gold tags are not directly comparable  However, to measure how these systems might perform without gold tags, we run three recent state-of-the-art systems in our experimental setting: UR-A E-DMV BIBREF36 , Neural E-DMV BIBREF11 , and CRF Autoencoder (CRFAE) BIBREF37   We use unsupervised POS tags (induced from our Markov-structured model) in place of gold tags  We also train basic DMV on gold tags and include several state-of-the-art results on gold tags as reference points 
As shown in Table TABREF39 , our approach is able to improve over the Gaussian DMV by 4 8 points on length INLINEFORM0 and 4 8 points on all lengths, which suggests the additional latent embedding layer and neural projector are helpful  The proposed approach yields, to the best of our knowledge, state-of-the-art performance without gold POS annotation and without sentence-internal boundary information  DMV, UR-A E-DMV, Neural E-DMV, and CRFAE suffer a large decrease in performance when trained on unsupervised tags – an effect also seen in previous work BIBREF29 , BIBREF34   Since our approach induces latent POS tags jointly with dependency trees, it may be able to learn POS clusters that are more amenable to grammar induction than the unsupervised tags  We observe that CRFAE underperforms its gold-tag counterpart substantially  This may largely be a result of the model's reliance on prior linguistic rules that become unavailable when gold POS tag types are unknown  Many extensions to DMV can be considered orthogonal to our approach – they essentially focus on improving the syntax model  It is possible that incorporating these more sophisticated syntax models into our approach may lead to further improvements 
Sensitivity Analysis
In the above experiments we initialize the structured syntax components with the pre-trained Gaussian or discrete baseline, which is shown as a useful technique to help train our deep models  We further study the results with fully random initialization  In the POS tagging experiment, we report the results in Table TABREF48   While the performance with 4 layers is comparable to the pre-trained Gaussian initialization, deeper projections (8 or 16 layers) result in a dramatic drop in performance  This suggests that the structured syntax model with very deep projections is difficult to train from scratch, and a simpler projection might be a good compromise in the random initialization setting 
Different from the Markov prior in POS tagging experiments, our parsing model seems to be quite sensitive to the initialization  For example, directed accuracy of our approach on sentences of length INLINEFORM0 is below 40 0 with random initialization  This is consistent with previous work that has noted the importance of careful initialization for DMV-based models such as the commonly used harmonic initializer BIBREF2   However, it is not straightforward to apply the harmonic initializer for DMV directly in our model without using some kind of pre-training since we do not observe gold POS 
We investigate the effect of the choice of pre-trained embedding on performance while using our approach  To this end, we additionally include results using fastText embeddings BIBREF41 – which, in contrast with skip-gram embeddings, include character-level information  We set the context windows size to 1 and the dimension size to 100 as in the skip-gram training, while keeping other parameters set to their defaults  These results are summarized in Table TABREF50 and Table TABREF51   While fastText embeddings lead to reduced performance with our model, our approach still yields an improvement over the Gaussian baseline with the new observed embeddings space 
Qualitative Analysis of Embeddings
We perform qualitative analysis to understand how the latent embeddings help induce syntactic structures  First we filter out low-frequency words and punctuations in WSJ, and visualize the rest words (10k) with t-SNE BIBREF42 under different embeddings  We assign each word with its most likely gold POS tags in WSJ and color them according to the gold POS tags 
For our Markov-structured model, we have displayed the embedding space in Figure SECREF5 , where the gold POS clusters are well-formed  Further, we present five example target words and their five nearest neighbors in terms of cosine similarity  As shown in Table TABREF53 , the skip-gram embedding captures both semantic and syntactic aspects to some degree, yet our embeddings are able to focus especially on the syntactic aspects of words, in an unsupervised fashion without using any extra morphological information 
In Figure FIGREF54 we depict the learned latent embeddings with the DMV-structured syntax model  Unlike the Markov structure, the DMV structure maps a large subset of singular and plural nouns to the same overlapping region  However, two clusters of singular and plural nouns are actually separated  We inspect the two clusters and the overlapping region in Figure FIGREF54 , it turns out that the nouns in the separated clusters are words that can appear as subjects and, therefore, for which verb agreement is important to model  In contrast, the nouns in the overlapping region are typically objects  This demonstrates that the latent embeddings are focusing on aspects of language that are specifically important for modeling dependency without ever having seen examples of dependency parses  Some previous work has deliberately created embeddings to capture different notions of similarity BIBREF43 , BIBREF44 , while they use extra morphology or dependency annotations to guide the embedding learning, our approach provides a potential alternative to create new embeddings that are guided by structured syntax model, only using unlabeled text corpora 
Related Work
Our approach is related to flow-based generative models, which are first described in NICE BIBREF16 and have recently received more attention BIBREF17 , BIBREF19 , BIBREF18   This relevant work mostly adopts simple (e g  Gaussian) and fixed priors and does not attempt to learn interpretable latent structures  Another related generative model class is variational auto-encoders (VAEs) BIBREF45 that optimize a lower bound on the marginal data likelihood, and can be extended to learn latent structures BIBREF46 , BIBREF47   Against the flow-based models, VAEs remove the invertibility constraint but sacrifice the merits of exact inference and exact log likelihood computation, which potentially results in optimization challenges BIBREF48   Our approach can also be viewed in connection with generative adversarial networks (GANs) BIBREF49 that is a likelihood-free framework to learn implicit generative models  However, it is non-trivial for a gradient-based method like GANs to propagate gradients through discrete structures 
Conclusion
In this work, we define a novel generative approach to leverage continuous word representations for unsupervised learning of syntactic structure  Experiments on both POS induction and unsupervised dependency parsing tasks demonstrate the effectiveness of our proposed approach  Future work might explore more sophisticated invertible projections, or recurrent projections that jointly transform the entire input sequence 

ground_truth: English

English 

English