prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Were other baselines tested to compare with the neural baseline?
Context 1: Introduction
Privacy policies are the documents which disclose the ways in which a company gathers, uses, shares and manages a user's data  As legal documents, they function using the principle of notice and choice BIBREF0, where companies post their policies, and theoretically, users read the policies and decide to use a company's products or services only if they find the conditions outlined in its privacy policy acceptable  Many legal jurisdictions around the world accept this framework, including the United States and the European Union BIBREF1, BIBREF2  However, the legitimacy of this framework depends upon users actually reading and understanding privacy policies to determine whether company practices are acceptable to them BIBREF3  In practice this is seldom the case BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10  This is further complicated by the highly individual and nuanced compromises that users are willing to make with their data BIBREF11, discouraging a `one-size-fits-all' approach to notice of data practices in privacy documents 
With devices constantly monitoring our environment, including our personal space and our bodies, lack of awareness of how our data is being used easily leads to problematic situations where users are outraged by information misuse, but companies insist that users have consented  The discovery of increasingly egregious uses of data by companies, such as the scandals involving Facebook and Cambridge Analytica BIBREF12, have further brought public attention to the privacy concerns of the internet and ubiquitous computing  This makes privacy a well-motivated application domain for NLP researchers, where advances in enabling users to quickly identify the privacy issues most salient to them can potentially have large real-world impact 
[1]https://play google com/store/apps/details?id=com gotokeep keep intl [2]https://play google com/store/apps/details?id=com viber voip [3]A question might not have any supporting evidence for an answer within the privacy policy 
Motivated by this need, we contribute PrivacyQA, a corpus consisting of 1750 questions about the contents of privacy policies, paired with over 3500 expert annotations  The goal of this effort is to kickstart the development of question-answering methods for this domain, to address the (unrealistic) expectation that a large population should be reading many policies per day  In doing so, we identify several understudied challenges to our ability to answer these questions, with broad implications for systems seeking to serve users' information-seeking intent  By releasing this resource, we hope to provide an impetus to develop systems capable of language understanding in this increasingly important domain 
Related Work
Prior work has aimed to make privacy policies easier to understand  Prescriptive approaches towards communicating privacy information BIBREF21, BIBREF22, BIBREF23 have not been widely adopted by industry  Recently, there have been significant research effort devoted to understanding privacy policies by leveraging NLP techniques BIBREF24, BIBREF25, BIBREF26, BIBREF27, BIBREF28, especially by identifying specific data practices within a privacy policy  We adopt a personalized approach to understanding privacy policies, that allows users to query a document and selectively explore content salient to them  Most similar is the PolisisQA corpus BIBREF29, which examines questions users ask corporations on Twitter  Our approach differs in several ways: 1) The PrivacyQA dataset is larger, containing 10x as many questions and answers  2) Answers are formulated by domain experts with legal training  3) PrivacyQA includes diverse question types, including unanswerable and subjective questions 
Our work is also related to reading comprehension in the open domain, which is frequently based upon Wikipedia passages BIBREF16, BIBREF17, BIBREF15, BIBREF30 and news articles BIBREF20, BIBREF31, BIBREF32  Table TABREF4 presents the desirable attributes our dataset shares with past approaches  This work is also tied into research in applying NLP approaches to legal documents BIBREF33, BIBREF34, BIBREF35, BIBREF36, BIBREF37, BIBREF38, BIBREF39  While privacy policies have legal implications, their intended audience consists of the general public rather than individuals with legal expertise  This arrangement is problematic because the entities that write privacy policies often have different goals than the audience  feng2015applying, tan-EtAl:2016:P16-1 examine question answering in the insurance domain, another specialized domain similar to privacy, where the intended audience is the general public 
Data Collection
We describe the data collection methodology used to construct PrivacyQA  With the goal of achieving broad coverage across application types, we collect privacy policies from 35 mobile applications representing a number of different categories in the Google Play Store  One of our goals is to include both policies from well-known applications, which are likely to have carefully-constructed privacy policies, and lesser-known applications with smaller install bases, whose policies might be considerably less sophisticated  Thus, setting 5 million installs as a threshold, we ensure each category includes applications with installs on both sides of this threshold  All policies included in the corpus are in English, and were collected before April 1, 2018, predating many companies' GDPR-focused BIBREF41 updates  We leave it to future studies BIBREF42 to look at the impact of the GDPR (e g , to what extent GDPR requirements contribute to making it possible to provide users with more informative answers, and to what extent their disclosures continue to omit issues that matter to users) 
Data Collection ::: Crowdsourced Question Elicitation
The intended audience for privacy policies consists of the general public  This informs the decision to elicit questions from crowdworkers on the contents of privacy policies  We choose not to show the contents of privacy policies to crowdworkers, a procedure motivated by a desire to avoid inadvertent biases BIBREF43, BIBREF44, BIBREF45, BIBREF46, BIBREF47, and encourage crowdworkers to ask a variety of questions beyond only asking questions based on practices described in the document 
Instead, crowdworkers are presented with public information about a mobile application available on the Google Play Store including its name, description and navigable screenshots  Figure FIGREF9 shows an example of our user interface  Crowdworkers are asked to imagine they have access to a trusted third-party privacy assistant, to whom they can ask any privacy question about a given mobile application  We use the Amazon Mechanical Turk platform and recruit crowdworkers who have been conferred “master” status and are located within the United States of America  Turkers are asked to provide five questions per mobile application, and are paid $2 per assignment, taking ~eight minutes to complete the task 
Data Collection ::: Answer Selection
To identify legally sound answers, we recruit seven experts with legal training to construct answers to Turker questions  Experts identify relevant evidence within the privacy policy, as well as provide meta-annotation on the question's relevance, subjectivity, OPP-115 category BIBREF49, and how likely any privacy policy is to contain the answer to the question asked 
Data Collection ::: Analysis
Table TABREF17 presents aggregate statistics of the PrivacyQA dataset  1750 questions are posed to our imaginary privacy assistant over 35 mobile applications and their associated privacy documents  As an initial step, we formulate the problem of answering user questions as an extractive sentence selection task, ignoring for now background knowledge, statistical data and legal expertise that could otherwise be brought to bear  The dataset is partitioned into a training set featuring 27 mobile applications and 1350 questions, and a test set consisting of 400 questions over 8 policy documents  This ensures that documents in training and test splits are mutually exclusive  Every question is answered by at least one expert  In addition, in order to estimate annotation reliability and provide for better evaluation, every question in the test set is answered by at least two additional experts 
Table TABREF14 describes the distribution over first words of questions posed by crowdworkers  We also observe low redundancy in the questions posed by crowdworkers over each policy, with each policy receiving ~49 94 unique questions despite crowdworkers independently posing questions  Questions are on average 8 4 words long  As declining to answer a question can be a legally sound response but is seldom practically useful, answers to questions where a minority of experts abstain to answer are filtered from the dataset  Privacy policies are ~3000 words long on average  The answers to the question asked by the users typically have ~100 words of evidence in the privacy policy document 
Data Collection ::: Analysis ::: Categories of Questions
Questions are organized under nine categories from the OPP-115 Corpus annotation scheme BIBREF49:
First Party Collection/Use: What, why and how information is collected by the service provider
Third Party Sharing/Collection: What, why and how information shared with or collected by third parties
Data Security: Protection measures for user information
Data Retention: How long user information will be stored
User Choice/Control: Control options available to users
User Access, Edit and Deletion: If/how users can access, edit or delete information
Policy Change: Informing users if policy information has been changed
International and Specific Audiences: Practices pertaining to a specific group of users
Other: General text, contact information or practices not covered by other categories 
For each question, domain experts indicate one or more relevant OPP-115 categories  We mark a category as relevant to a question if it is identified as such by at least two annotators  If no such category exists, the category is marked as `Other' if atleast one annotator has identified the `Other' category to be relevant  If neither of these conditions is satisfied, we label the question as having no agreement  The distribution of questions in the corpus across OPP-115 categories is as shown in Table TABREF16  First party and third party related questions are the largest categories, forming nearly 66 4% of all questions asked to the privacy assistant 
Data Collection ::: Analysis ::: Answer Validation
When do experts disagree? We would like to analyze the reasons for potential disagreement on the annotation task, to ensure disagreements arise due to valid differences in opinion rather than lack of adequate specification in annotation guidelines  It is important to note that the annotators are experts rather than crowdworkers  Accordingly, their judgements can be considered valid, legally-informed opinions even when their perspectives differ  For the sake of this question we randomly sample 100 instances in the test data and analyze them for likely reasons for disagreements  We consider a disagreement to have occurred when more than one expert does not agree with the majority consensus  By disagreement we mean there is no overlap between the text identified as relevant by one expert and another 
We find that the annotators agree on the answer for 74% of the questions, even if the supporting evidence they identify is not identical i e full overlap  They disagree on the remaining 26%  Sources of apparent disagreement correspond to situations when different experts: have differing interpretations of question intent (11%) (for example, when a user asks 'who can contact me through the app', the questions admits multiple interpretations, including seeking information about the features of the app, asking about first party collection/use of data or asking about third party collection/use of data), identify different sources of evidence for questions that ask if a practice is performed or not (4%), have differing interpretations of policy content (3%), identify a partial answer to a question in the privacy policy (2%) (for example, when the user asks `who is allowed to use the app' a majority of our annotators decline to answer, but the remaining annotators highlight partial evidence in the privacy policy which states that children under the age of 13 are not allowed to use the app), and other legitimate sources of disagreement (6%) which include personal subjective views of the annotators (for example, when the user asks `is my DNA information used in any way other than what is specified', some experts consider the boilerplate text of the privacy policy which states that it abides to practices described in the policy document as sufficient evidence to answer this question, whereas others do not) 
Experimental Setup
We evaluate the ability of machine learning methods to identify relevant evidence for questions in the privacy domain  We establish baselines for the subtask of deciding on the answerability (§SECREF33) of a question, as well as the overall task of identifying evidence for questions from policies (§SECREF37)  We describe aspects of the question that can render it unanswerable within the privacy domain (§SECREF41) 
Experimental Setup ::: Answerability Identification Baselines
We define answerability identification as a binary classification task, evaluating model ability to predict if a question can be answered, given a question in isolation  This can serve as a prior for downstream question-answering  We describe three baselines on the answerability task, and find they considerably improve performance over a majority-class baseline 
SVM: We define 3 sets of features to characterize each question  The first is a simple bag-of-words set of features over the question (SVM-BOW), the second is bag-of-words features of the question as well as length of the question in words (SVM-BOW + LEN), and lastly we extract bag-of-words features, length of the question in words as well as part-of-speech tags for the question (SVM-BOW + LEN + POS)  This results in vectors of 200, 201 and 228 dimensions respectively, which are provided to an SVM with a linear kernel 
CNN: We utilize a CNN neural encoder for answerability prediction  We use GloVe word embeddings BIBREF50, and a filter size of 5 with 64 filters to encode questions 
BERT: BERT BIBREF51 is a bidirectional transformer-based language-model BIBREF52  We fine-tune BERT-base on our binary answerability identification task with a learning rate of 2e-5 for 3 epochs, with a maximum sequence length of 128 
Experimental Setup ::: Privacy Question Answering
Our goal is to identify evidence within a privacy policy for questions asked by a user  This is framed as an answer sentence selection task, where models identify a set of evidence sentences from all candidate sentences in each policy 
Experimental Setup ::: Privacy Question Answering ::: Evaluation Metric
Our evaluation metric for answer-sentence selection is sentence-level F1, implemented similar to BIBREF30, BIBREF16  Precision and recall are implemented by measuring the overlap between predicted sentences and sets of gold-reference sentences  We report the average of the maximum F1 from each n$-$1 subset, in relation to the heldout reference 
Experimental Setup ::: Privacy Question Answering ::: Baselines
We describe baselines on this task, including a human performance baseline 
No-Answer Baseline (NA) : Most of the questions we receive are difficult to answer in a legally-sound way on the basis of information present in the privacy policy  We establish a simple baseline to quantify the effect of identifying every question as unanswerable 
Word Count Baseline : To quantify the effect of using simple lexical matching to answer the questions, we retrieve the top candidate policy sentences for each question using a word count baseline BIBREF53, which counts the number of question words that also appear in a sentence  We include the top 2, 3 and 5 candidates as baselines 
BERT: We implement two BERT-based baselines BIBREF51 for evidence identification  First, we train BERT on each query-policy sentence pair as a binary classification task to identify if the sentence is evidence for the question or not (Bert)  We also experiment with a two-stage classifier, where we separately train the model on questions only to predict answerability  At inference time, if the answerable classifier predicts the question is answerable, the evidence identification classifier produces a set of candidate sentences (Bert + Unanswerable) 
Human Performance: We pick each reference answer provided by an annotator, and compute the F1 with respect to the remaining references, as described in section 4 2 1  Each reference answer is treated as the prediction, and the remaining n-1 answers are treated as the gold reference  The average of the maximum F1 across all reference answers is computed as the human baseline 
Results and Discussion
The results of the answerability baselines are presented in Table TABREF31, and on answer sentence selection in Table TABREF32  We observe that bert exhibits the best performance on a binary answerability identification task  However, most baselines considerably exceed the performance of a majority-class baseline  This suggests considerable information in the question, indicating it's possible answerability within this domain 
Table TABREF32 describes the performance of our baselines on the answer sentence selection task  The No-answer (NA) baseline performs at 28 F1, providing a lower bound on performance at this task  We observe that our best-performing baseline, Bert + Unanswerable achieves an F1 of 39 8  This suggest that bert is capable of making some progress towards answering questions in this difficult domain, while still leaving considerable headroom for improvement to reach human performance  Bert + Unanswerable performance suggests that incorporating information about answerability can help in this difficult domain  We examine this challenging phenomena of unanswerability further in Section  
Results and Discussion ::: Error Analysis
Disagreements are analyzed based on the OPP-115 categories of each question (Table TABREF34)  We compare our best performing BERT variant against the NA model and human performance  We observe significant room for improvement across all categories of questions but especially for first party, third party and data retention categories 
We analyze the performance of our strongest BERT variant, to identify classes of errors and directions for future improvement (Table 8)  We observe that a majority of answerability mistakes made by the BERT model are questions which are in fact answerable, but are identified as unanswerable by BERT  We observe that BERT makes 124 such mistakes on the test set  We collect expert judgments on relevance, subjectivity , silence and information about how likely the question is to be answered from the privacy policy from our experts  We find that most of these mistakes are relevant questions  However many of them were identified as subjective by the annotators, and at least one annotator marked 19 of these questions as having no answer within the privacy policy  However, only 6 of these questions were unexpected or do not usually have an answer in privacy policies  These findings suggest that a more nuanced understanding of answerability might help improve model performance in his challenging domain 
Results and Discussion ::: What makes Questions Unanswerable?
We further ask legal experts to identify potential causes of unanswerability of questions  This analysis has considerable implications  While past work BIBREF17 has treated unanswerable questions as homogeneous, a question answering system might wish to have different treatments for different categories of `unanswerable' questions  The following factors were identified to play a role in unanswerability:
Incomprehensibility: If a question is incomprehensible to the extent that its meaning is not intelligible 
Relevance: Is this question in the scope of what could be answered by reading the privacy policy 
Ill-formedness: Is this question ambiguous or vague  An ambiguous statement will typically contain expressions that can refer to multiple potential explanations, whereas a vague statement carries a concept with an unclear or soft definition 
Silence: Other policies answer this type of question but this one does not 
Atypicality: The question is of a nature such that it is unlikely for any policy policy to have an answer to the question 
Our experts attempt to identify the different `unanswerable' factors for all 573 such questions in the corpus  4 18% of the questions were identified as being incomprehensible (for example, `any difficulties to occupy the privacy assistant')  Amongst the comprehendable questions, 50% were identified as likely to have an answer within the privacy policy, 33 1% were identified as being privacy-related questions but not within the scope of a privacy policy (e g , 'has Viber had any privacy breaches in the past?') and 16 9% of questions were identified as completely out-of-scope (e g , `'will the app consume much space?')  In the questions identified as relevant, 32% were ill-formed questions that were phrased by the user in a manner considered vague or ambiguous  Of the questions that were both relevant as well as `well-formed', 95 7% of the questions were not answered by the policy in question but it was reasonable to expect that a privacy policy would contain an answer  The remaining 4 3% were described as reasonable questions, but of a nature generally not discussed in privacy policies  This suggests that the answerability of questions over privacy policies is a complex issue, and future systems should consider each of these factors when serving user's information seeking intent 
We examine a large-scale dataset of “natural” unanswerable questions BIBREF54 based on real user search engine queries to identify if similar unanswerability factors exist  It is important to note that these questions have previously been filtered, according to a criteria for bad questions defined as “(questions that are) ambiguous, incomprehensible, dependent on clear false presuppositions, opinion-seeking, or not clearly a request for factual information ” Annotators made the decision based on the content of the question without viewing the equivalent Wikipedia page  We randomly sample 100 questions from the development set which were identified as unanswerable, and find that 20% of the questions are not questions (e g , “all I want for christmas is you mariah carey tour”)  12% of questions are unlikely to ever contain an answer on Wikipedia, corresponding closely to our atypicality category  3% of questions are unlikely to have an answer anywhere (e g , `what guides Santa home after he has delivered presents?')  7% of questions are incomplete or open-ended (e g , `the south west wind blows across nigeria between')  3% of questions have an unresolvable coreference (e g , `how do i get to Warsaw Missouri from here')  4% of questions are vague, and a further 7% have unknown sources of error  2% still contain false presuppositions (e g , `what is the only fruit that does not have seeds?') and the remaining 42% do not have an answer within the document  This reinforces our belief that though they have been understudied in past work, any question answering system interacting with real users should expect to receive such unanticipated and unanswerable questions 
Conclusion
We present PrivacyQA, the first significant corpus of privacy policy questions and more than 3500 expert annotations of relevant answers  The goal of this work is to promote question-answering research in the specialized privacy domain, where it can have large real-world impact  Strong neural baselines on PrivacyQA achieve a performance of only 39 8 F1 on this corpus, indicating considerable room for future research  Further, we shed light on several important considerations that affect the answerability of questions  We hope this contribution leads to multidisciplinary efforts to precisely understand user intent and reconcile it with information in policy documents, from both the privacy and NLP communities 
Acknowledgements
This research was supported in part by grants from the National Science Foundation Secure and Trustworthy Computing program (CNS-1330596, CNS-1330214, CNS-15-13957, CNS-1801316, CNS-1914486, CNS-1914444) and a DARPA Brandeis grant on Personalized Privacy Assistants (FA8750-15-2-0277)  The US Government is authorized to reproduce and distribute reprints for Governmental purposes not withstanding any copyright notation  The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the NSF, DARPA, or the US Government  The authors would like to extend their gratitude to Elias Wright, Gian Mascioli, Kiara Pillay, Harrison Kay, Eliel Talo, Alexander Fagella and N  Cameron Russell for providing their valuable expertise and insight to this effort  The authors are also grateful to Eduard Hovy, Lorrie Cranor, Florian Schaub, Joel Reidenberg, Aditya Potukuchi and Igor Shalyminov for helpful discussions related to this work, and to the three anonymous reviewers of this draft for their constructive feedback  Finally, the authors would like to thank all crowdworkers who consented to participate in this study 
Context 2: Introduction
Users of photo-sharing websites such as Flickr often provide short textual descriptions in the form of tags to help others find the images  With the availability of GPS systems in current electronic devices such as smartphones, latitude and longitude coordinates are nowadays commonly made available as well  The tags associated with such georeferenced photos often describe the location where these photos were taken, and Flickr can thus be regarded as a source of environmental information  The use of Flickr for modelling urban environments has already received considerable attention  For instance, various approaches have been proposed for modelling urban regions BIBREF0 , and for identifying points-of-interest BIBREF1 and itineraries BIBREF2 , BIBREF3   However, the usefulness of Flickr for characterizing the natural environment, which is the focus of this paper, is less well-understood 
Many recent studies have highlighted that Flickr tags capture valuable ecological information, which can be used as a complementary source to more traditional sources  To date, however, ecologists have mostly used social media to conduct manual evaluations of image content with little automated exploitation of the associated tags BIBREF4 , BIBREF5 , BIBREF6   One recent exception is BIBREF7 , where bag-of-words representations derived from Flickr tags were found to give promising result for predicting a range of different environemental phenomena 
Our main hypothesis in this paper is that by using vector space embeddings instead of bag-of-words representations, the ecological information which is implicitly captured by Flickr tags can be utilized in a more effective way  Vector space embeddings are representations in which the objects from a given domain are encoded using relatively low-dimensional vectors  They have proven useful in natural language processing, especially for encoding word meaning BIBREF8 , BIBREF9 , and in machine learning more generally  In this paper, we are interested in the use of such representations for modelling geographic locations  Our main motivation for using vector space embeddings is that they allow us to integrate the textual information we get from Flickr with available structured information in a very natural way  To this end, we rely on an adaptation of the GloVe word embedding model BIBREF9 , but rather than learning word vectors, we learn vectors representing locations  Similar to how the representation of a word in GloVe is determined by the context words surrounding it, the representation of a location in our model is determined by the tags of the photos that have been taken near that location  To incorporate numerical features from structured environmental datasets (e g  average temperature), we associate with each such feature a linear mapping that can be used to predict that feature from a given location vector  This is inspired by the fact that salient properties of a given domain can often be modelled as directions in vector space embeddings BIBREF10 , BIBREF11 , BIBREF12   Finally, evidence from categorical datasets (e g  land cover types) is taken into account by requiring that locations belonging to the same category are represented using similar vectors, similar to how semantic types are sometimes modelled in the context of knowledge graph embedding BIBREF13  
While our point-of-departure is a standard word embedding model, we found that the off-the-shelf GloVe model performed surprisingly poorly, meaning that a number of modifications are needed to achieve good results  Our main findings are as follows  First, given that the number of tags associated with a given location can be quite small, it is important to apply some kind of spatial smoothing, i e  the importance of a given tag for a given location should not only depend on the occurrences of the tag at that location, but also on its occurrences at nearby locations  To this end, we use a formulation which is based on spatially smoothed version of pointwise mutual information  Second, given the wide diversity in the kind of information that is covered by Flickr tags, we find that term selection is in some cases critical to obtain vector spaces that capture the relevant aspects of geographic locations  For instance, many tags on Flickr refer to photography related terms, which we would normally not want to affect the vector representation of a given location  Finally, even with these modifications, vector space embeddings learned from Flickr tags alone are sometimes outperformed by bag-of-words representations  However, our vector space embeddings lead to substantially better predictions in cases where structured (scientific) information is also taken into account  In this sense, the main value of using vector space embeddings in this context is not so much about abstracting away from specific tag usages, but rather about the fact that such representations allow us to integrate numerical and categorical features in a much more natural way than is possible with bag-of-words representations 
The remainder of this paper is organized as follows  In the next section, we provide a discussion of existing work  Section SECREF3 then presents our model for embedding geographic locations from Flickr tags and structured data  Next, in Section SECREF4 we provide a detailed discussion about the experimental results  Finally, Section SECREF5 summarizes our conclusions 
Vector space embeddings
The use of low-dimensional vector space embeddings for representing objects has already proven effective in a large number of applications, including natural language processing (NLP), image processing, and pattern recognition  In the context of NLP, the most prominent example is that of word embeddings, which represent word meaning using vectors of typically around 300 dimensions  A large number of different methods for learning such word embeddings have already been proposed, including Skip-gram and the Continuous Bag-of-Words (CBOW) model BIBREF8 , GloVe BIBREF9 , and fastText BIBREF14   They have been applied effectively in many downstream NLP tasks such as sentiment analysis BIBREF15 , part of speech tagging BIBREF16 , BIBREF17 , and text classification BIBREF18 , BIBREF19   The model we consider in this paper builds on GloVe, which was designed to capture linear regularities of word-word co-occurrence  In GloVe, there are two word vectors INLINEFORM0 and INLINEFORM1 for each word in the vocabulary, which are learned by minimizing the following objective: DISPLAYFORM0
where INLINEFORM0 is the number of times that word INLINEFORM1 appears in the context of word INLINEFORM2 , INLINEFORM3 is the vocabulary size, INLINEFORM4 is the target word bias, INLINEFORM5 is the context word bias  The weighting function INLINEFORM6 is used to limit the impact of rare terms  It is defined as 1 if INLINEFORM7 and as INLINEFORM8 otherwise, where INLINEFORM9 is usually fixed to 100 and INLINEFORM10 to 0 75  Intuitively, the target word vectors INLINEFORM11 correspond to the actual word representations which we would like to find, while the context word vectors INLINEFORM12 model how occurrences of INLINEFORM13 in the context of a given word INLINEFORM14 affect the representation of this latter word  In this paper we will use a similar model, which will however be aimed at learning location vectors instead of the target word vectors 
Beyond word embeddings, various methods have been proposed for learning vector space representations from structured data such as knowledge graphs BIBREF20 , BIBREF21 , BIBREF22 , social networks BIBREF23 , BIBREF24 and taxonomies BIBREF25 , BIBREF26   The idea of combining a word embedding model with structured information has also been explored by several authors, for example to improve the word embeddings based on information coming from knowledge graphs BIBREF27 , BIBREF28   Along similar lines, various lexicons have been used to obtain word embeddings that are better suited at modelling sentiment BIBREF15 and antonymy BIBREF29 , among others  The method proposed by BIBREF30 imposes the condition that words that belong to the same semantic category are closer together than words from different categories, which is somewhat similar in spirit to how we will model categorical datasets in our model 
Embeddings for geographic information
The problem of representing geographic locations using embeddings has also attracted some attention  An early example is BIBREF31 , which used principal component analysis and stacked autoencoders to learn low-dimensional vector representations of city neighbourhoods based on census data  They use these representations to predict attributes such as crime, which is not included in the given census data, and find that in most of the considered evaluation tasks, the low-dimensional vector representations lead to more faithful predictions than the original high-dimensional census data 
Some existing works combine word embedding models with geographic coordinates  For example, in BIBREF32 an approach is proposed to learn word embeddings based on the assumption that words which tend to be used in the same geographic locations are likely to be similar  Note that their aim is dual to our aim in this paper: while they use geographic location to learn word vectors, we use textual descriptions to learn vectors representing geographic locations 
Several methods also use word embedding models to learn representations of Points-of-Interest (POIs) that can be used for predicting user visits BIBREF33 , BIBREF34 , BIBREF35   These works use the machinery of existing word embedding models to learn POI representations, intuitively by letting sequences of POI visits by a user play the role of sequences of words in a sentence  In other words, despite the use of word embedding models, many of these approaches do not actually consider any textual information  For example, in BIBREF34 the Skip-gram model is utilized to create a global pattern of users' POIs  Each location was treated as a word and the other locations visited before or after were treated as context words  They then use a pair-wise ranking loss BIBREF36 which takes into account the user's location visit frequency to personalize the location recommendations  The methods of BIBREF34 were extended in BIBREF35 to use a temporal embedding and to take more account of geographic context, in particular the distances between preferred and non-preferred neighboring POIs, to create a “geographically hierarchical pairwise preference ranking model”  Similarly, in BIBREF37 the CBOW model was trained with POI data  They ordered POIs spatially within the traffic-based zones of urban areas  The ordering was used to generate characteristic vectors of POI types  Zone vectors represented by averaging the vectors of the POIs contained in them, were then used as features to predict land use types  In the CrossMap method BIBREF38 they learned embeddings for spatio-temporal hotspots obtained from social media data of locations, times and text  In one form of embedding, intended to enable reconstruction of records, neighbourhood relations in space and time were encoded by averaging hotspots in a target location's spatial and temporal neighborhoods  They also proposed a graph-based embedding method with nodes of location, time and text  The concatenation of the location, time and text vectors were then used as features to predict peoples' activities in urban environments  Finally, in BIBREF39 , a method is proposed that uses the Skip-gram model to represent POI types, based on the intuition that the vector representing a given POI type should be predictive of the POI types that found near places of that type 
Our work is different from these studies, as our focus is on representing locations based on a given text description of that location (in the form of Flickr tags), along with numerical and categorical features from scientific datasets 
Analyzing Flickr tags
Many studies have focused on analyzing Flickr tags to extract useful information in domains such as linguistics BIBREF40 , geography BIBREF0 , BIBREF41 , and ecology BIBREF42 , BIBREF7 , BIBREF43   Most closely related to our work, BIBREF7 found that the tags of georeferenced Flickr photos can effectively supplement traditional scientific environmental data in tasks such as predicting climate features, land cover, species occurrence, and human assessments of scenicness  To encode locations, they simply combine a bag-of-words representation of geographically nearby tags with a feature vector that encodes associated structured scientific data  They found that the predictive value of Flickr tags is roughly on a par with that of the scientific datasets, and that combining both types of information leads to significantly better results than using either of them alone  As we show in this paper, however, their straightforward way of combining both information sources, by concatenating the two types of feature vectors, is far from optimal 
Despite the proven importance of Flickr tags, the problem of embedding Flickr tags has so far received very limited attention  To the best of our knowledge, BIBREF44 is the only work that generated embeddings for Flickr tags  However, their focus was on learning embeddings that capture word meaning (being evaluated on word similarity tasks), whereas we use such embeddings as part of our method for representing locations 
Model Description
In this section, we introduce our embedding model, which combines Flickr tags and structured scientific information to represent a set of locations INLINEFORM0   The proposed model has the following form: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are parameters to control the importance of each component in the model  Component INLINEFORM2 will be used to constrain the representation of the locations based on their textual description (i e  Flickr tags), INLINEFORM3 will be used to constrain the representation of the locations based on their numerical features, and INLINEFORM4 will impose the constraint that locations belonging to the same category should be close together in the space  We will discuss each of these components in more detail in the following sections 
Tag Based Location Embedding
Many of the tags associated with Flickr photos describe characteristics of the places where these photos were taken BIBREF45 , BIBREF46 , BIBREF47   For example, tags may correspond to place names (e g  Brussels, England, Scandinavia), landmarks (e g  Eiffel Tower, Empire State Building) or land cover types (e g  mountain, forest, beach)  To allow us to build location models using such tags, we collected the tags and meta-data of 70 million Flickr photos with coordinates in Europe (which is the region our experiments will focus on), all of which were uploaded to Flickr before the end of September 2015  In this section we first explain how tags can be weighted to obtain bag-of-words representations of locations from Flickr  Subsequently we describe a tag selection method, which will allow us to specialize the embedding depending on which aspects of the considered locations are of interest, after which we discuss the actual embedding model 
Tag weighting  Let INLINEFORM0 be a set of geographic locations, each characterized by latitude and longitude coordinates  To generate a bag-of-words representation of a given location, we have to weight the relevance of each tag to that location  To this end, we have followed the weighting scheme from BIBREF7 , which combines a Gaussian kernel (to model spatial proximity) with Positive Pointwise Mutual Information (PPMI) BIBREF48 , BIBREF49  
Let us write INLINEFORM0 for the set of users who have assigned tag INLINEFORM1 to a photo with coordinates near INLINEFORM2   To assess how relevant INLINEFORM3 is to the location INLINEFORM4 , the number of times INLINEFORM5 occurs in photos near INLINEFORM6 is clearly an important criterion  However, rather than simply counting the number of occurrences within some fixed radius, we use a Gaussian kernel to weight the tag occurrences according to their distance from that location: INLINEFORM7
where the threshold INLINEFORM0 is assumed to be fixed, INLINEFORM1 is the location of a Flickr photo, INLINEFORM2 is the Haversine distance, and we will assume that the bandwidth parameter INLINEFORM3 is set to INLINEFORM4   A tag occurrence is counted only once for all photos by the same user at the same location, which is important to reduce the impact of bulk uploading  The value INLINEFORM5 reflects how frequent tag INLINEFORM6 is near location INLINEFORM7 , but it does not yet take into account the total number of tag occurrences near INLINEFORM8 , nor how popular the tag INLINEFORM9 is overall  To measure how strongly tag INLINEFORM10 is associated with location INLINEFORM11 , we use PPMI, which is a commonly used measure of association in natural language processing  However, rather than estimating PPMI scores from term frequencies, we will use the INLINEFORM12 values instead: INLINEFORM13
where: INLINEFORM0
with INLINEFORM0 the set of all tags, and INLINEFORM1 the set of locations 
Tag selection  Inspired by BIBREF50 , we use a term selection method in order to focus on the tags that are most important for the tasks that we want to consider and reduce the impact of tags that might relate only to a given individual or a group of users  In particular, we obtained good results with a method based on Kullback-Leibler (KL) divergence, which is based on BIBREF51   Let INLINEFORM0 be a set of (mutually exclusive) properties of locations in which we are interested (e g  land cover categories)  For the ease of presentation, we will identify INLINEFORM1 with the set of locations that have the corresponding property  Then, we select tags from INLINEFORM2 that maximize the following score: INLINEFORM3
where INLINEFORM0 is the probability that a photo with tag INLINEFORM1 has a location near INLINEFORM2 and INLINEFORM3 is the probability that an arbitrary tag occurrence is assigned to a photo near a location in INLINEFORM4   Since INLINEFORM5 often has to be estimated from a small number of tag occurrences, it is estimated using Bayesian smoothing: INLINEFORM6
where INLINEFORM0 is a parameter controlling the amount of smoothing, which will be tuned in the experiments  On the other hand, for INLINEFORM1 we can simply use a maximum likelihood estimation: INLINEFORM2
Location embedding  We now want to find a vector INLINEFORM0 for each location INLINEFORM1 such that similar locations are represented using similar vectors  To achieve this, we use a close variant of the GloVe model, where tag occurrences are treated as context words of geographic locations  In particular, with each location INLINEFORM2 we associate a vector INLINEFORM3 and with each tag INLINEFORM4 we associate a vector INLINEFORM5 and a bias term INLINEFORM6 , and consider the following objective (which in our full model ( EQREF7 ) will be combined with components that are derived from the structured information): INLINEFORM7
Note how tags play the role of the context words in the GloVe model, while instead of learning target word vectors we now learn location vectors  In contrast to GloVe, our objective does not directly refer to co-occurrence statistics, but instead uses the INLINEFORM0 scores  One important consequence of this is that we can also consider pairs INLINEFORM1 for which INLINEFORM2 does not occur in INLINEFORM3 at all; such pairs are usually called negative examples  While they cannot be used in the standard GloVe model, some authors have already reported that introducing negative examples in variants of GloVe can lead to an improvement BIBREF52   In practice, evaluating the full objective above would not be computationally feasible, as we may need to consider millions of locations and millions of tags  Therefore, rather than considering all tags in INLINEFORM4 for the inner summation, we only consider those tags that appear at least once near location INLINEFORM5 together with a sample of negative examples 
Structured Environmental Data
There is a wide variety of structured data that can be used to describe locations  In this work, we have restricted ourselves to the same datasets as BIBREF7   These include nine (real-valued) numerical features, which are latitude, longitude, elevation, population, and five climate related features (avg  temperature, avg  precipitation, avg  solar radiation, avg  wind speed, and avg  water vapor pressure)  In addition, 180 categorical features were used, which are CORINE land cover classes at level 1 (5 classes), level 2 (15 classes) and level 3 (44 classes) and 116 soil types (SoilGrids)  Note that each location should belong to exactly 4 categories: one CORINE class at each of the three levels and a soil type 
Numerical features  Numerical features can be treated similarly to the tag occurrences, i e  we will assume that the value of a given numerical feature can be predicted from the location vectors using a linear mapping  In particular, for each numerical feature INLINEFORM0 we consider a vector INLINEFORM1 and a bias term INLINEFORM2 , and the following objective: INLINEFORM3
where we write INLINEFORM0 for set of all numerical features and INLINEFORM1 is the value of feature INLINEFORM2 for location INLINEFORM3 , after z-score normalization 
Categorical features  To take into account the categorical features, we impose the constraint that locations belonging to the same category should be close together in the space  To formalize this, we represent each category type INLINEFORM0 as a vector INLINEFORM1 , and consider the following objective: INLINEFORM2
Evaluation Tasks
We will use the method from BIBREF7 as our main baseline  This will allow us to directly evaluate the effectiveness of embeddings for the considered problem, since we have used the same structured datasets and same tag weighting scheme  For this reason, we will also follow their evaluation methodology  In particular, we will consider three evaluation tasks:
Predicting the distribution of 100 species across Europe, using the European network of nature protected sites Natura 2000 dataset as ground truth  For each of these species, a binary classification problem is considered  The set of locations INLINEFORM0 is defined as the 26,425 distinct sites occurring in the dataset 
Predicting soil type, again each time treating the task as a binary classification problem, using the same set of locations INLINEFORM0 as in the species distribution experiments  For these experiments, none of the soil type features are used for generating the embeddings 
Predicting CORINE land cover classes at levels 1, 2 and level 3, each time treating the task as a binary classification problem, using the same set of locations INLINEFORM0 as in the species distribution experiments  For these experiments, none of the CORINE features are used for generating the embeddings 
In addition, we will also consider the following regression tasks:
Predicting 5 climate related features: the average precipitation, temperature, solar radiation, water vapor pressure, and wind speed  We again use the same set of locations INLINEFORM0 as for species distribution in this experiment  None of the climate features is used for constructing the embeddings for this experiment 
Predicting people's subjective opinions of landscape beauty in Britain, using the crowdsourced dataset from the ScenicOrNot website as ground truth  The set INLINEFORM0 is chosen as the set of locations of 191 605 rated locations from the ScenicOrNot dataset for which at least one georeferenced Flickr photo exists within a 1 km radius 
Experimental Setup
In all experiments, we use Support Vector Machines (SVMs) for classification problems and Support Vector Regression (SVR) for regression problems to make predictions from our representations of geographic locations  In both cases, we used the SVM INLINEFORM0 implementation BIBREF53   For each experiment, the set of locations INLINEFORM1 was split into two-thirds for training, one-sixth for testing, and one-sixth for tuning the parameters  All embedding models are learned with Adagrad using 30 iterations  The number of dimensions is chosen for each experiment from INLINEFORM2 based on the tuning data  For the parameters of our model in Equation EQREF7 , we considered values of INLINEFORM3 from {0 1, 0 01, 0 001, 0 0001} and values of INLINEFORM4 from {1, 10, 100, 1000, 10 000, 100 000}  To compute KL divergence, we need to determine a set of classes INLINEFORM5 for each experiment  For classification problems, we can simply consider the given categories, but for the regression problems we need to define such classes by discretizing the numerical values  For the scenicness experiments, we considered scores 3 and 7 as cut-off points, leading to three classes (i e  less than 3, between 3 and 7, and above 7)  Similarly, for each climate related features, we consider two cut-off values for discretization: 5 and 15 for average temperature, 50 and 100 for average precipitation, 10 000 and 17 000 for average solar radiation, 0 7 and 1 for average water vapor pressure, and 3 and 5 for wind speed  The smoothing parameter INLINEFORM6 was selected among INLINEFORM7 based on the tuning data  In all experiments where term selection is used, we select the top 100 000 tags  We fixed the radius INLINEFORM8 at 1km when counting the number of tag occurrences  Finally, we set the number of negative examples as 10 times the number of positive examples for each location, but with a cap at 1000 negative examples in each region for computational reasons  We tune all parameters with respect to the F1 score for the classification tasks, and Spearman INLINEFORM9 for the regression tasks 
Variants and Baseline Methods
We will refer to our model as EGEL (Embedding GEographic Locations), and will consider the following variants  EGEL-Tags only uses the information from the Flickr tags (i e  component INLINEFORM0 ), without using any negative examples and without feature selection  EGEL-Tags+NS is similar to EGEL-Tags but with the addition of negative examples  EGEL-KL(Tags+NS) additionally considers term selection  EGEL-All is our full method, i e  it additionally uses the structured information  We also consider the following baselines  BOW-Tags represents locations using a bag-of-words representation, using the same tag weighting as the embedding model  BOW-KL(Tags) uses the same representation but after term selection, using the same KL-based method as the embedding model  BOW-All combines the bag-of-words representation with the structured information, encoded as proposed in BIBREF7   GloVe uses the objective from the original GloVe model for learning location vectors, i e  this variant differs from EGEL-Tags in that instead of INLINEFORM1 we use the number of co-occurrences of tag INLINEFORM2 near location INLINEFORM3 , measured as INLINEFORM4  
Results and Discussion
We present our results for the binary classification tasks in Tables TABREF23 – TABREF24 in terms of average precision, average recall and macro average F1 score  The results of the regression tasks are reported in Tables TABREF25 and TABREF29 in terms of the mean absolute error between the predicted and actual scores, as well as the Spearman INLINEFORM0 correlation between the rankings induced by both sets of scores  It can be clearly seen from the results that our proposed method (EGEL-All) can effectively integrate Flickr tags with the available structured information  It outperforms the baselines for all the considered tasks  Furthermore, note that the PPMI-based weighting in EGEL-Tags consistently outperforms GloVe and that both the addition of negative examples and term selection lead to further improvements  The use of term selection leads to particularly substantial improvements for the regression problems 
While our experimental results confirm the usefulness of embeddings for predicting environmental features, this is only consistently the case for the variants that use both the tags and the structured datasets  In particular, comparing BOW-Tags with EGEL-Tags, we sometimes see that the former achieves the best results  While this might seem surprising, it is in accordance with the findings in BIBREF54 , BIBREF38 , among others, where it was also found that bag-of-words representations can sometimes lead to surprisingly effective baselines  Interestingly, we note that in all cases where EGEL-KL(Tags+NS) performs worse than BOW-Tags, we also find that BOW-KL(Tags) performs worse than BOW-Tags  This suggests that for these tasks there is a very large variation in the kind of tags that can inform the prediction model, possibly including e g  user-specific tags  Some of the information captured by such highly specific but rare tags is likely to be lost in the embedding 
To further analyze the difference in performance between BoW representations and embeddings, Figure TABREF29 compares the performance of the GloVe model with the bag-of-words model for predicting place scenicness, as a function of the number of tag occurrences at the considered locations  What is clearly noticeable in Figure TABREF29 is that GloVe performs better than the bag-of-words model for large corpora and worse for smaller corpora  This issue has been alleviated in our embedding method by the addition of negative examples 
 Conclusions
In this paper, we have proposed a model to learn geographic location embeddings using Flickr tags, numerical environmental features, and categorical information  The experimental results show that our model can integrate Flickr tags with structured information in a more effective way than existing methods, leading to substantial improvements over baseline methods on various prediction tasks about the natural environment 
Acknowledgments
Shelan Jeawak has been sponsored by HCED Iraq  Steven Schockaert has been supported by ERC Starting Grant 637277 
Context 3: Introduction
Conventional automatic speech recognition (ASR) systems typically consist of several independently learned components: an acoustic model to predict context-dependent sub-phoneme states (senones) from audio, a graph structure to map senones to phonemes, and a pronunciation model to map phonemes to words  Hybrid systems combine hidden Markov models to model state dependencies with neural networks to predict states BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   Newer approaches such as end-to-end (E2E) systems reduce the overall complexity of the final system 
Our research builds on prior work that has explored using time-delay neural networks (TDNN), other forms of convolutional neural networks, and Connectionist Temporal Classification (CTC) loss BIBREF4 , BIBREF5 , BIBREF6   We took inspiration from wav2letter BIBREF6 , which uses 1D-convolution layers  Liptchinsky et al  BIBREF7 improved wav2letter by increasing the model depth to 19 convolutional layers and adding Gated Linear Units (GLU) BIBREF8 , weight normalization BIBREF9 and dropout 
By building a deeper and larger capacity network, we aim to demonstrate that we can match or outperform non end-to-end models on the LibriSpeech and 2000hr Fisher+Switchboard tasks  Like wav2letter, our architecture, Jasper, uses a stack of 1D-convolution layers, but with ReLU and batch normalization BIBREF10   We find that ReLU and batch normalization outperform other activation and normalization schemes that we tested for convolutional ASR  As a result, Jasper's architecture contains only 1D convolution, batch normalization, ReLU, and dropout layers – operators highly optimized for training and inference on GPUs 
It is possible to increase the capacity of the Jasper model by stacking these operations  Our largest version uses 54 convolutional layers (333M parameters), while our small model uses 34 (201M parameters)  We use residual connections to enable this level of depth  We investigate a number of residual options and propose a new residual connection topology we call Dense Residual (DR) 
Integrating our best acoustic model with a Transformer-XL BIBREF11 language model allows us to obtain new state-of-the-art (SOTA) results on LibriSpeech BIBREF12 test-clean of 2 95% WER and SOTA results among end-to-end models on LibriSpeech test-other  We show competitive results on Wall Street Journal (WSJ), and 2000hr Fisher+Switchboard (F+S)  Using only greedy decoding without a language model we achieve 3 86% WER on LibriSpeech test-clean 
This paper makes the following contributions:
Jasper Architecture
Jasper is a family of end-to-end ASR models that replace acoustic and pronunciation models with a convolutional neural network  Jasper uses mel-filterbank features calculated from 20ms windows with a 10ms overlap, and outputs a probability distribution over characters per frame  Jasper has a block architecture: a Jasper INLINEFORM0 x INLINEFORM1 model has INLINEFORM2 blocks, each with INLINEFORM3 sub-blocks  Each sub-block applies the following operations: a 1D-convolution, batch norm, ReLU, and dropout  All sub-blocks in a block have the same number of output channels 
Each block input is connected directly into the last sub-block via a residual connection  The residual connection is first projected through a 1x1 convolution to account for different numbers of input and output channels, then through a batch norm layer  The output of this batch norm layer is added to the output of the batch norm layer in the last sub-block  The result of this sum is passed through the activation function and dropout to produce the output of the sub-block 
The sub-block architecture of Jasper was designed to facilitate fast GPU inference  Each sub-block can be fused into a single GPU kernel: dropout is not used at inference-time and is eliminated, batch norm can be fused with the preceding convolution, ReLU clamps the result, and residual summation can be treated as a modified bias term in this fused operation 
All Jasper models have four additional convolutional blocks: one pre-processing and three post-processing  See Figure FIGREF7 and Table TABREF8 for details 
We also build a variant of Jasper, Jasper Dense Residual (DR)  Jasper DR follows DenseNet BIBREF15 and DenseRNet BIBREF16 , but instead of having dense connections within a block, the output of a convolution block is added to the inputs of all the following blocks  While DenseNet and DenseRNet concatenates the outputs of different layers, Jasper DR adds them in the same way that residuals are added in ResNet  As explained below, we find addition to be as effective as concatenation 
Normalization and Activation
In our study, we evaluate performance of models with:
3 types of normalization: batch norm BIBREF10 , weight norm BIBREF9 , and layer norm BIBREF17
3 types of rectified linear units: ReLU, clipped ReLU (cReLU), and leaky ReLU (lReLU)
2 types of gated units: gated linear units (GLU) BIBREF8 , and gated activation units (GAU) BIBREF18
All experiment results are shown in Table TABREF15   We first experimented with a smaller Jasper5x3 model to pick the top 3 settings before training on larger Jasper models  We found that layer norm with GAU performed the best on the smaller model  Layer norm with ReLU and batch norm with ReLU came second and third in our tests  Using these 3, we conducted further experiments on a larger Jasper10x4  For larger models, we noticed that batch norm with ReLU outperformed other choices  Thus, leading us to decide on batch normalization and ReLU for our architecture 
During batching, all sequences are padded to match the longest sequence  These padded values caused issues when using layer norm  We applied a sequence mask to exclude padding values from the mean and variance calculation  Further, we computed mean and variance over both the time dimension and channels similar to the sequence-wise normalization proposed by Laurent et al  BIBREF19   In addition to masking layer norm, we additionally applied masking prior to the convolution operation, and masking the mean and variance calculation in batch norm  These results are shown in Table TABREF16   Interestingly, we found that while masking before convolution gives a lower WER, using masks for both convolutions and batch norm results in worse performance 
As a final note, we found that training with weight norm was very unstable leading to exploding activations 
Residual Connections
For models deeper than Jasper 5x3, we observe consistently that residual connections are necessary for training to converge  In addition to the simple residual and dense residual model described above, we investigated DenseNet BIBREF15 and DenseRNet BIBREF16 variants of Jasper  Both connect the outputs of each sub-block to the inputs of following sub-blocks within a block  DenseRNet, similar to Dense Residual, connects the output of each output of each block to the input of all following blocks  DenseNet and DenseRNet combine residual connections using concatenation whereas Residual and Dense Residual use addition  We found that Dense Residual and DenseRNet perform similarly with each performing better on specific subsets of LibriSpeech  We decided to use Dense Residual for subsequent experiments  The main reason is that due to concatenation, the growth factor for DenseNet and DenseRNet requires tuning for deeper models whereas Dense Residual simply just repeats a sub-blocks 
Language Model
A language model (LM) is a probability distribution over arbitrary symbol sequences INLINEFORM0 such that more likely sequences are assigned high probabilities  LMs are frequently used to condition beam search  During decoding, candidates are evaluated using both acoustic scores and LM scores  Traditional N-gram LMs have been augmented with neural LMs in recent work BIBREF20 , BIBREF21 , BIBREF22  
We experiment with statistical N-gram language models BIBREF23 and neural Transformer-XL BIBREF11 models  Our best results use acoustic and word-level N-gram language models to generate a candidate list using beam search with a width of 2048  Next, an external Transformer-XL LM rescores the final list  All LMs were trained on datasets independently from acoustic models  We show results with the neural LM in our Results section  We observed a strong correlation between the quality of the neural LM (measured by perplexity) and WER as shown in Figure FIGREF20  
NovoGrad
For training, we use either Stochastic Gradient Descent (SGD) with momentum or our own NovoGrad, an optimizer similar to Adam BIBREF14 , except that its second moments are computed per layer instead of per weight  Compared to Adam, it reduces memory consumption and we find it to be more numerically stable 
At each step INLINEFORM0 , NovoGrad computes the stochastic gradient INLINEFORM1 following the regular forward-backward pass  Then the second-order moment INLINEFORM2 is computed for each layer INLINEFORM3 similar to ND-Adam BIBREF27 : DISPLAYFORM0
The second-order moment INLINEFORM0 is used to re-scale gradients INLINEFORM1 before calculating the first-order moment INLINEFORM2 : DISPLAYFORM0
If L2-regularization is used, a weight decay INLINEFORM0 is added to the re-scaled gradient (as in AdamW BIBREF28 ): DISPLAYFORM0
Finally, new weights are computed using the learning rate INLINEFORM0 : DISPLAYFORM0
Using NovoGrad instead of SGD with momentum, we decreased the WER on dev-clean LibriSpeech from 4 00% to 3 64%, a relative improvement of 9% for Jasper DR 10x5  We will further analyze NovoGrad in forthcoming work 
Results
We evaluate Jasper across a number of datasets in various domains  In all experiments, we use dropout and weight decay as regularization  At training time, we use speed perturbation with fixed +/-10% BIBREF29 for LibriSpeech  For WSJ and Hub5'00, we use a random speed perturbation factor between [-10%, 10%] as each utterance is fed into the model  All models have been trained on NVIDIA DGX-1 in mixed precision BIBREF30 using OpenSeq2Seq BIBREF31   Source code, training configurations, and pretrained models are available 
Read Speech
We evaluated the performance of Jasper on two read speech datasets: LibriSpeech and Wall Street Journal (WSJ)  For LibriSpeech, we trained Jasper DR 10x5 using our NovoGrad optimizer for 400 epochs  We achieve SOTA performance on the test-clean subset and SOTA among end-to-end speech recognition models on test-other 
We trained a smaller Jasper 10x3 model with SGD with momentum optimizer for 400 epochs on a combined WSJ dataset (80 hours): LDC93S6A (WSJ0) and LDC94S13A (WSJ1)  The results are provided in Table TABREF29  
Conversational Speech
We also evaluate the Jasper model's performance on a conversational English corpus  The Hub5 Year 2000 (Hub5'00) evaluation (LDC2002S09, LDC2005S13) is widely used in academia  It is divided into two subsets: Switchboard (SWB) and Callhome (CHM)  The training data for both the acoustic and language models consisted of the 2000hr Fisher+Switchboard training data (LDC2004S13, LDC2005S13, LDC97S62)  Jasper DR 10x5 was trained using SGD with momentum for 50 epochs  We compare to other models trained using the same data and report Hub5'00 results in Table TABREF31  
We obtain good results for SWB  However, there is work to be done to improve WER on harder tasks such as CHM 
Conclusions
We have presented a new family of neural architectures for end-to-end speech recognition  Inspired by wav2letter's convolutional approach, we build a deep and scalable model, which requires a well-designed residual topology, effective regularization, and a strong optimizer  As our architecture studies demonstrated, a combination of standard components leads to SOTA results on LibriSpeech and competitive results on other benchmarks  Our Jasper architecture is highly efficient for training and inference, and serves as a good baseline approach on top of which to explore more sophisticated regularization, data augmentation, loss functions, language models, and optimization strategies  We are interested to see if our approach can continue to scale to deeper models and larger datasets 

ground_truth: SVM, No-Answer Baseline (NA) , Word Count Baseline, Human Performance

No-Answer Baseline (NA), Word Count Baseline, Human Performance

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what are the existing approaches?
Context 4: Introduction
Data annotation is a major bottleneck for the application of supervised learning approaches to many problems  As a result, unsupervised methods that learn directly from unlabeled data are increasingly important  For tasks related to unsupervised syntactic analysis, discrete generative models have dominated in recent years – for example, for both part-of-speech (POS) induction BIBREF0 , BIBREF1 and unsupervised dependency parsing BIBREF2 , BIBREF3 , BIBREF4   While similar models have had success on a range of unsupervised tasks, they have mostly ignored the apparent utility of continuous word representations evident from supervised NLP applications BIBREF5 , BIBREF6   In this work, we focus on leveraging and explicitly representing continuous word embeddings within unsupervised models of syntactic structure 
Pre-trained word embeddings from massive unlabeled corpora offer a compact way of injecting a prior notion of word similarity into models that would otherwise treat words as discrete, isolated categories  However, the specific properties of language captured by any particular embedding scheme can be difficult to control, and, further, may not be ideally suited to the task at hand  For example, pre-trained skip-gram embeddings BIBREF7 with small context window size are found to capture the syntactic properties of language well BIBREF8 , BIBREF9   However, if our goal is to separate syntactic categories, this embedding space is not ideal – POS categories correspond to overlapping interspersed regions in the embedding space, evident in Figure SECREF4  
In our approach, we propose to learn a new latent embedding space as a projection of pre-trained embeddings (depicted in Figure SECREF5 ), while jointly learning latent syntactic structure – for example, POS categories or syntactic dependencies  To this end, we introduce a new generative model (shown in Figure FIGREF6 ) that first generates a latent syntactic representation (e g  a dependency parse) from a discrete structured prior (which we also call the “syntax model”), then, conditioned on this representation, generates a sequence of latent embedding random variables corresponding to each word, and finally produces the observed (pre-trained) word embeddings by projecting these latent vectors through a parameterized non-linear function  The latent embeddings can be jointly learned with the structured syntax model in a completely unsupervised fashion 
By choosing an invertible neural network as our non-linear projector, and then parameterizing our model in terms of the projection's inverse, we are able to derive tractable exact inference and marginal likelihood computation procedures so long as inference is tractable in the underlying syntax model  In sec:learn-with-inv we show that this derivation corresponds to an alternate view of our approach whereby we jointly learn a mapping of observed word embeddings to a new embedding space that is more suitable for the syntax model, but include an additional Jacobian regularization term to prevent information loss 
Recent work has sought to take advantage of word embeddings in unsupervised generative models with alternate approaches BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12   BIBREF9 build an HMM with Gaussian emissions on observed word embeddings, but they do not attempt to learn new embeddings  BIBREF10 , BIBREF11 , and BIBREF12 extend HMM or dependency model with valence (DMV) BIBREF2 with multinomials that use word (or tag) embeddings in their parameterization  However, they do not represent the embeddings as latent variables 
In experiments, we instantiate our approach using both a Markov-structured syntax model and a tree-structured syntax model – specifically, the DMV  We evaluate on two tasks: part-of-speech (POS) induction and unsupervised dependency parsing without gold POS tags  Experimental results on the Penn Treebank BIBREF13 demonstrate that our approach improves the basic HMM and DMV by a large margin, leading to the state-of-the-art results on POS induction, and state-of-the-art results on unsupervised dependency parsing in the difficult training scenario where neither gold POS annotation nor punctuation-based constraints are available 
Model
As an illustrative example, we first present a baseline model for Markov syntactic structure (POS induction) that treats a sequence of pre-trained word embeddings as observations  Then, we propose our novel approach, again using Markov structure, that introduces latent word embedding variables and a neural projector  Lastly, we extend our approach to more general syntactic structures 
Example: Gaussian HMM
We start by describing the Gaussian hidden Markov model introduced by BIBREF9 , which is a locally normalized model with multinomial transitions and Gaussian emissions  Given a sentence of length INLINEFORM0 , we denote the latent POS tags as INLINEFORM1 , observed (pre-trained) word embeddings as INLINEFORM2 , transition parameters as INLINEFORM3 , and Gaussian emission parameters as INLINEFORM4   The joint distribution of data and latent variables factors as:
DISPLAYFORM0
where INLINEFORM0 is the multinomial transition probability and INLINEFORM1 is the multivariate Gaussian emission probability 
While the observed word embeddings do inform this model with a notion of word similarity – lacking in the basic multinomial HMM – the Gaussian emissions may not be sufficiently flexible to separate some syntactic categories in the complex pre-trained embedding space – for example the skip-gram embedding space as visualized in Figure SECREF4 where different POS categories overlap  Next we introduce a new approach that adds flexibility to the emission distribution by incorporating new latent embedding variables 
Markov Structure with Neural Projector
To flexibly model observed embeddings and yield a new representation space that is more suitable for the syntax model, we propose to cascade a neural network as a projection function, deterministically transforming the simple space defined by the Gaussian HMM to the observed embedding space  We denote the latent embedding of the INLINEFORM0 word in a sentence as INLINEFORM1 , and the neural projection function as INLINEFORM2 , parameterized by INLINEFORM3   In the case of sequential Markov structure, our new model corresponds to the following generative process:
For each time step INLINEFORM0 ,
[noitemsep, leftmargin=*]
Draw the latent state INLINEFORM0
Draw the latent embedding INLINEFORM0
Deterministically produce embedding
INLINEFORM0
The graphical model is depicted in Figure FIGREF6   The deterministic projection can also be viewed as sampling each observation from a point mass at INLINEFORM0   The joint distribution of our model is: DISPLAYFORM0
where INLINEFORM0 is a conditional Gaussian distribution, and INLINEFORM1 is the Dirac delta function centered at INLINEFORM2 : DISPLAYFORM0
General Structure with Neural Projector
Our approach can be applied to a broad family of structured syntax models  We denote latent embedding variables as INLINEFORM0 , discrete latent variables in the syntax model as INLINEFORM1 ( INLINEFORM2 ), where INLINEFORM3 are conditioned to generate INLINEFORM4   The joint probability of our model factors as:
DISPLAYFORM0
where INLINEFORM0 represents the probability of the syntax model, and can encode any syntactic structure – though, its factorization structure will determine whether inference is tractable in our full model  As shown in Figure FIGREF6 , we focus on two syntax models for syntactic analysis in this paper  The first is Markov-structured, which we use for POS induction, and the second is DMV-structured, which we use to learn dependency parses without supervision 
The marginal data likelihood of our model is: DISPLAYFORM0
While the discrete variables INLINEFORM0 can be marginalized out with dynamic program in many cases, it is generally intractable to marginalize out the latent continuous variables, INLINEFORM1 , for an arbitrary projection INLINEFORM2 in Eq  ( EQREF17 ), which means inference and learning may be difficult  In sec:opt, we address this issue by constraining INLINEFORM3 to be invertible, and show that this constraint enables tractable exact inference and marginal likelihood computation 
Learning & Inference
In this section, we introduce an invertibility condition for our neural projector to tackle the optimization challenge  Specifically, we constrain our neural projector with two requirements: (1) INLINEFORM0 and (2) INLINEFORM1 exists  Invertible transformations have been explored before in independent components analysis BIBREF14 , gaussianization BIBREF15 , and deep density models BIBREF16 , BIBREF17 , BIBREF18 , for unstructured data  Here, we generalize this style of approach to structured learning, and augment it with discrete latent variables ( INLINEFORM2 )  Under the invertibility condition, we derive a learning algorithm and give another view of our approach revealed by the objective function  Then, we present the architecture of a neural projector we use in experiments: a volume-preserving invertible neural network proposed by BIBREF16 for independent components estimation 
Learning with Invertibility
For ease of exposition, we explain the learning algorithm in terms of Markov structure without loss of generality  As shown in Eq  ( EQREF17 ), the optimization challenge in our approach comes from the intractability of the marginalized emission factor INLINEFORM0   If we can marginalize out INLINEFORM1 and compute INLINEFORM2 , then the posterior and marginal likelihood of our Markov-structured model can be computed with the forward-backward algorithm  We can apply Eq  ( EQREF14 ) and obtain : INLINEFORM3
By using the change of variable rule to the integration, which allows the integration variable INLINEFORM0 to be replaced by INLINEFORM1 , the marginal emission factor can be computed in closed-form when the invertibility condition is satisfied: DISPLAYFORM0
where INLINEFORM0 is a conditional Gaussian distribution, INLINEFORM1 is the Jacobian matrix of function INLINEFORM2 at INLINEFORM3 , and INLINEFORM4 represents the absolute value of its determinant  This Jacobian term is nonzero and differentiable if and only if INLINEFORM5 exists 
Eq  ( EQREF19 ) shows that we can directly calculate the marginal emission distribution INLINEFORM0   Denote the marginal data likelihood of Gaussian HMM as INLINEFORM1 , then the log marginal data likelihood of our model can be directly written as: DISPLAYFORM0
where INLINEFORM0 represents the new sequence of embeddings after applying INLINEFORM1 to each INLINEFORM2   Eq  ( EQREF20 ) shows that the training objective of our model is simply the Gaussian HMM log likelihood with an additional Jacobian regularization term  From this view, our approach can be seen as equivalent to reversely projecting the data through INLINEFORM3 to another manifold INLINEFORM4 that is directly modeled by the Gaussian HMM, with a regularization term  Intuitively, we optimize the reverse projection INLINEFORM5 to modify the INLINEFORM6 space, making it more appropriate for the syntax model  The Jacobian regularization term accounts for the volume expansion or contraction behavior of the projection  Maximizing it can be thought of as preventing information loss  In the extreme case, the Jacobian determinant is equal to zero, which means the projection is non-invertible and thus information is being lost through the projection  Such “information preserving” regularization is crucial during optimization, otherwise the trivial solution of always projecting data to the same single point to maximize likelihood is viable 
More generally, for an arbitrary syntax model the data likelihood of our approach is: DISPLAYFORM0
If the syntax model itself allows for tractable inference and marginal likelihood computation, the same dynamic program can be used to marginalize out INLINEFORM0   Therefore, our joint model inherits the tractability of the underlying syntax model 
Invertible Volume-Preserving Neural Net
For the projection we can use an arbitrary invertible function, and given the representational power of neural networks they seem a natural choice  However, calculating the inverse and Jacobian of an arbitrary neural network can be difficult, as it requires that all component functions be invertible and also requires storage of large Jacobian matrices, which is memory intensive  To address this issue, several recent papers propose specially designed invertible networks that are easily trainable yet still powerful BIBREF16 , BIBREF17 , BIBREF19   Inspired by these works, we use the invertible transformation proposed by BIBREF16 , which consists of a series of “coupling layers”  This architecture is specially designed to guarantee a unit Jacobian determinant (and thus the invertibility property) 
From Eq  ( EQREF22 ) we know that only INLINEFORM0 is required for accomplishing learning and inference; we never need to explicitly construct INLINEFORM1   Thus, we directly define the architecture of INLINEFORM2   As shown in Figure FIGREF24 , the nonlinear transformation from the observed embedding INLINEFORM3 to INLINEFORM4 represents the first coupling layer  The input in this layer is partitioned into left and right halves of dimensions, INLINEFORM5 and INLINEFORM6 , respectively  A single coupling layer is defined as: DISPLAYFORM0
where INLINEFORM0 is the coupling function and can be any nonlinear form  This transformation satisfies INLINEFORM1 , and BIBREF16 show that its Jacobian matrix is triangular with all ones on the main diagonal  Thus the Jacobian determinant is always equal to one (i e  volume-preserving) and the invertibility condition is naturally satisfied 
To be sufficiently expressive, we compose multiple coupling layers as suggested in BIBREF16   Specifically, we exchange the role of left and right half vectors at each layer as shown in Figure FIGREF24   For instance, from INLINEFORM0 to INLINEFORM1 the left subset INLINEFORM2 is unchanged, while from INLINEFORM3 to INLINEFORM4 the right subset INLINEFORM5 remains the same  Also note that composing multiple coupling layers does not change the volume-preserving and invertibility properties  Such a sequence of invertible transformations from the data space INLINEFORM6 to INLINEFORM7 is also called normalizing flow BIBREF20  
Experiments
In this section, we first describe our datasets and experimental setup  We then instantiate our approach with Markov and DMV-structured syntax models, and report results on POS tagging and dependency grammar induction respectively  Lastly, we analyze the learned latent embeddings 
Data
For both POS tagging and dependency parsing, we run experiments on the Wall Street Journal (WSJ) portion of the Penn Treebank  To create the observed data embeddings, we train skip-gram word embeddings BIBREF7 that are found to capture syntactic properties well when trained with small context window BIBREF8 , BIBREF9   Following BIBREF9 , the dimensionality INLINEFORM0 is set to 100, and the training context window size is set to 1 to encode more syntactic information  The skip-gram embeddings are trained on the one billion word language modeling benchmark dataset BIBREF21 in addition to the WSJ corpus 
General Experimental Setup
For the neural projector, we employ rectified networks as coupling function INLINEFORM0 following BIBREF16   We use a rectified network with an input layer, one hidden layer, and linear output units, the number of hidden units is set to the same as the number of input units  The number of coupling layers are varied as 4, 8, 16 for both tasks

ground_truth: BOW-Tags, BOW-KL(Tags), BOW-All, GloVe

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what were the baselines?
Context 4:  We optimize marginal data likelihood directly using Adam BIBREF22   For both tasks in the fully unsupervised setting, we do not tune the hyper-parameters using supervised data 
Unsupervised POS tagging
For unsupervised POS tagging, we use a Markov-structured syntax model in our approach, which is a popular structure for unsupervised tagging tasks BIBREF9 , BIBREF10  
Following existing literature, we train and test on the entire WSJ corpus (49208 sentences, 1M tokens)  We use 45 tag clusters, the number of POS tags that appear in WSJ corpus  We train the discrete HMM and the Gaussian HMM BIBREF9 as baselines  For the Gaussian HMM, mean vectors of Gaussian emissions are initialized with the empirical mean of all word vectors with an additive noise  We assume diagonal covariance matrix for INLINEFORM0 and initialize it with the empirical variance of the word vectors  Following BIBREF9 , the covariance matrix is fixed during training  The multinomial probabilities are initialized as INLINEFORM1 , where INLINEFORM2   For our approach, we initialize the syntax model and Gaussian parameters with the pre-trained Gaussian HMM  The weights of layers in the rectified network are initialized from a uniform distribution with mean zero and a standard deviation of INLINEFORM3 , where INLINEFORM4 is the input dimension  We evaluate the performance of POS tagging with both Many-to-One (M-1) accuracy BIBREF23 and V-Measure (VM) BIBREF24   Given a model we found that the tagging performance is well-correlated with the training data likelihood, thus we use training data likelihood as a unsupervised criterion to select the trained model over 10 random restarts after training 50 epochs  We repeat this process 5 times and report the mean and standard deviation of performance 
We compare our approach with basic HMM, Gaussian HMM, and several state-of-the-art systems, including sophisticated HMM variants and clustering techniques with hand-engineered features  The results are presented in Table TABREF32   Through the introduced latent embeddings and additional neural projection, our approach improves over the Gaussian HMM by 5 4 points in M-1 and 5 6 points in VM  Neural HMM (NHMM) BIBREF10 is a baseline that also learns word representation jointly  Both their basic model and extended Conv version does not outperform the Gaussian HMM  Their best model incorporates another LSTM to model long distance dependency and breaks the Markov assumption, yet our approach still achieves substantial improvement over it without considering more context information  Moreover, our method outperforms the best published result that benefits from hand-engineered features BIBREF27 by 2 0 points on VM 
We found that most tagging errors happen in noun subcategories  Therefore, we do the one-to-one mapping between gold POS tags and induced clusters and plot the normalized confusion matrix of noun subcategories in Figure FIGREF35   The Gaussian HMM fails to identify “NN” and “NNS” correctly for most cases, and it often recognizes “NNPS” as “NNP”  In contrast, our approach corrects these errors well 
Unsupervised Dependency Parsing without gold POS tags
For the task of unsupervised dependency parse induction, we employ the Dependency Model with Valence (DMV) BIBREF2 as the syntax model in our approach  DMV is a generative model that defines a probability distribution over dependency parse trees and syntactic categories, generating tokens and dependencies in a head-outward fashion  While, traditionally, DMV is trained using gold POS tags as observed syntactic categories, in our approach, we treat each tag as a latent variable, as described in sec:general-neural 
Most existing approaches to this task are not fully unsupervised since they rely on gold POS tags following the original experimental setup for DMV  This is partially because automatically parsing from words is difficult even when using unsupervised syntactic categories BIBREF29   However, inducing dependencies from words alone represents a more realistic experimental condition since gold POS tags are often unavailable in practice  Previous work that has trained from words alone often requires additional linguistic constraints (like sentence internal boundaries) BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , acoustic cues BIBREF33 , additional training data BIBREF4 , or annotated data from related languages BIBREF34   Our approach is naturally designed to train on word embeddings directly, thus we attempt to induce dependencies without using gold POS tags or other extra linguistic information 
Like previous work we use sections 02-21 of WSJ corpus as training data and evaluate on section 23, we remove punctuations and train the models on sentences of length INLINEFORM0 , “head-percolation” rules BIBREF39 are applied to obtain gold dependencies for evaluation  We train basic DMV, extended DMV (E-DMV) BIBREF35 and Gaussian DMV (which treats POS tag as unknown latent variables and generates observed word embeddings directly conditioned on them following Gaussian distribution) as baselines  Basic DMV and E-DMV are trained with Viterbi EM BIBREF40 on unsupervised POS tags induced from our Markov-structured model described in sec:pos  Multinomial parameters of the syntax model in both Gaussian DMV and our model are initialized with the pre-trained DMV baseline  Other parameters are initialized in the same way as in the POS tagging experiment  The directed dependency accuracy (DDA) is used for evaluation and we report accuracy on sentences of length INLINEFORM1 and all lengths  We train the parser until training data likelihood converges, and report the mean and standard deviation over 20 random restarts 
Our model directly observes word embeddings and does not require gold POS tags during training  Thus, results from related work trained on gold tags are not directly comparable  However, to measure how these systems might perform without gold tags, we run three recent state-of-the-art systems in our experimental setting: UR-A E-DMV BIBREF36 , Neural E-DMV BIBREF11 , and CRF Autoencoder (CRFAE) BIBREF37   We use unsupervised POS tags (induced from our Markov-structured model) in place of gold tags  We also train basic DMV on gold tags and include several state-of-the-art results on gold tags as reference points 
As shown in Table TABREF39 , our approach is able to improve over the Gaussian DMV by 4 8 points on length INLINEFORM0 and 4 8 points on all lengths, which suggests the additional latent embedding layer and neural projector are helpful  The proposed approach yields, to the best of our knowledge, state-of-the-art performance without gold POS annotation and without sentence-internal boundary information  DMV, UR-A E-DMV, Neural E-DMV, and CRFAE suffer a large decrease in performance when trained on unsupervised tags – an effect also seen in previous work BIBREF29 , BIBREF34   Since our approach induces latent POS tags jointly with dependency trees, it may be able to learn POS clusters that are more amenable to grammar induction than the unsupervised tags  We observe that CRFAE underperforms its gold-tag counterpart substantially  This may largely be a result of the model's reliance on prior linguistic rules that become unavailable when gold POS tag types are unknown  Many extensions to DMV can be considered orthogonal to our approach – they essentially focus on improving the syntax model  It is possible that incorporating these more sophisticated syntax models into our approach may lead to further improvements 
Sensitivity Analysis
In the above experiments we initialize the structured syntax components with the pre-trained Gaussian or discrete baseline, which is shown as a useful technique to help train our deep models  We further study the results with fully random initialization  In the POS tagging experiment, we report the results in Table TABREF48   While the performance with 4 layers is comparable to the pre-trained Gaussian initialization, deeper projections (8 or 16 layers) result in a dramatic drop in performance  This suggests that the structured syntax model with very deep projections is difficult to train from scratch, and a simpler projection might be a good compromise in the random initialization setting 
Different from the Markov prior in POS tagging experiments, our parsing model seems to be quite sensitive to the initialization  For example, directed accuracy of our approach on sentences of length INLINEFORM0 is below 40 0 with random initialization  This is consistent with previous work that has noted the importance of careful initialization for DMV-based models such as the commonly used harmonic initializer BIBREF2   However, it is not straightforward to apply the harmonic initializer for DMV directly in our model without using some kind of pre-training since we do not observe gold POS 
We investigate the effect of the choice of pre-trained embedding on performance while using our approach  To this end, we additionally include results using fastText embeddings BIBREF41 – which, in contrast with skip-gram embeddings, include character-level information  We set the context windows size to 1 and the dimension size to 100 as in the skip-gram training, while keeping other parameters set to their defaults  These results are summarized in Table TABREF50 and Table TABREF51   While fastText embeddings lead to reduced performance with our model, our approach still yields an improvement over the Gaussian baseline with the new observed embeddings space 
Qualitative Analysis of Embeddings
We perform qualitative analysis to understand how the latent embeddings help induce syntactic structures  First we filter out low-frequency words and punctuations in WSJ, and visualize the rest words (10k) with t-SNE BIBREF42 under different embeddings  We assign each word with its most likely gold POS tags in WSJ and color them according to the gold POS tags 
For our Markov-structured model, we have displayed the embedding space in Figure SECREF5 , where the gold POS clusters are well-formed  Further, we present five example target words and their five nearest neighbors in terms of cosine similarity  As shown in Table TABREF53 , the skip-gram embedding captures both semantic and syntactic aspects to some degree, yet our embeddings are able to focus especially on the syntactic aspects of words, in an unsupervised fashion without using any extra morphological information 
In Figure FIGREF54 we depict the learned latent embeddings with the DMV-structured syntax model  Unlike the Markov structure, the DMV structure maps a large subset of singular and plural nouns to the same overlapping region  However, two clusters of singular and plural nouns are actually separated  We inspect the two clusters and the overlapping region in Figure FIGREF54 , it turns out that the nouns in the separated clusters are words that can appear as subjects and, therefore, for which verb agreement is important to model  In contrast, the nouns in the overlapping region are typically objects  This demonstrates that the latent embeddings are focusing on aspects of language that are specifically important for modeling dependency without ever having seen examples of dependency parses  Some previous work has deliberately created embeddings to capture different notions of similarity BIBREF43 , BIBREF44 , while they use extra morphology or dependency annotations to guide the embedding learning, our approach provides a potential alternative to create new embeddings that are guided by structured syntax model, only using unlabeled text corpora 
Related Work
Our approach is related to flow-based generative models, which are first described in NICE BIBREF16 and have recently received more attention BIBREF17 , BIBREF19 , BIBREF18   This relevant work mostly adopts simple (e g  Gaussian) and fixed priors and does not attempt to learn interpretable latent structures  Another related generative model class is variational auto-encoders (VAEs) BIBREF45 that optimize a lower bound on the marginal data likelihood, and can be extended to learn latent structures BIBREF46 , BIBREF47   Against the flow-based models, VAEs remove the invertibility constraint but sacrifice the merits of exact inference and exact log likelihood computation, which potentially results in optimization challenges BIBREF48   Our approach can also be viewed in connection with generative adversarial networks (GANs) BIBREF49 that is a likelihood-free framework to learn implicit generative models  However, it is non-trivial for a gradient-based method like GANs to propagate gradients through discrete structures 
Conclusion
In this work, we define a novel generative approach to leverage continuous word representations for unsupervised learning of syntactic structure  Experiments on both POS induction and unsupervised dependency parsing tasks demonstrate the effectiveness of our proposed approach  Future work might explore more sophisticated invertible projections, or recurrent projections that jointly transform the entire input sequence 

ground_truth: Unanswerable

LF-MMI Attention
Seq2Seq 
RNN-T 
Char E2E LF-MMI 
Phone E2E LF-MMI 
CTC + Gram-CTC