prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the results from these proposed strategies?
Context 1: Introduction and Background
Many reinforcement learning algorithms are designed for relatively small discrete or continuous action spaces and so have trouble scaling  Text-adventure games—or interaction fictions—are simulations in which both an agents' state and action spaces are in textual natural language  An example of a one turn agent interaction in the popular text-game Zork1 can be seen in Fig  FIGREF1  Text-adventure games provide us with multiple challenges in the form of partial observability, commonsense reasoning, and a combinatorially-sized state-action space  Text-adventure games are structured as long puzzles or quests, interspersed with bottlenecks  The quests can usually be completed through multiple branching paths  However, games can also feature one or more bottlenecks  Bottlenecks are areas that an agent must pass through in order to progress to the next section of the game regardless of what path the agent has taken to complete that section of the quest BIBREF0  In this work, we focus on more effectively exploring this space and surpassing these bottlenecks—building on prior work that focuses on tackling the other problems 
Formally, we use the definition of text-adventure games as seen in BIBREF1 and BIBREF2  These games are partially observable Markov decision processes (POMDPs), represented as a 7-tuple of $\langle S,T,A,\Omega , O,R, \gamma \rangle $ representing the set of environment states, mostly deterministic conditional transition probabilities between states, the vocabulary or words used to compose text commands, observations returned by the game, observation conditional probabilities, reward function, and the discount factor respectively  For our purposes, understanding the exact state and action spaces we use in this work is critical and so we define each of these in relative depth 
Action-Space  To solve Zork1, the cannonical text-adventure games, requires the generation of actions consisting of up to five-words from a relatively modest vocabulary of 697 words recognized by the game’s parser  This results in $\mathcal {O}(697^5)={1 64e14}$ possible actions at every step  To facilitate text-adventure game playing, BIBREF2 introduce Jericho, a framework for interacting with text-games  They propose a template-based action space in which the agent first selects a template, consisting of an action verb and preposition, and then filling that in with relevant entities $($e g  $[get]$ $ [from] $ $)$  Zork1 has 237 templates, each with up to two blanks, yielding a template-action space of size $\mathcal {O}(237 \times 697^2)={1 15e8}$  This space is still far larger than most used by previous approaches applying reinforcement learning to text-based games 
State-Representation  Prior work has shown that knowledge graphs are effective in terms of dealing with the challenges of partial observability $($BIBREF3 BIBREF3; BIBREF4$)$  A knowledge graph is a set of 3-tuples of the form $\langle subject, relation, object \rangle $  These triples are extracted from the observations using Stanford's Open Information Extraction (OpenIE) BIBREF5  Human-made text-adventure games often contain relatively complex semi-structured information that OpenIE is not designed to parse and so they add additional rules to ensure that the correct information is parsed  The graph itself is more or less a map of the world, with information about objects' affordances and attributes linked to the rooms that they are place in a map  The graph also makes a distinction with respect to items that are in the agent's possession or in their immediate surrounding environment  An example of what the knowledge graph looks like and specific implementation details can be found in Appendix SECREF14 
BIBREF6 introduce the KG-A2C, which uses a knowledge graph based state-representation to aid in the section of actions in a combinatorially-sized action-space—specifically they use the knowledge graph to constrain the kinds of entities that can be filled in the blanks in the template action-space  They test their approach on Zork1, showing the combination of the knowledge graph and template action selection resulted in improvements over existing methods  They note that their approach reaches a score of 40 which corresponds to a bottleneck in Zork1 where the player is eaten by a “grue” (resulting in negative reward) if the player has not first lit a lamp  The lamp must be lit many steps after first being encountered, in a different section of the game; this action is necessary to continue exploring but doesn’t immediately produce any positive reward  That is, there is a long term dependency between actions that is not immediately rewarded, as seen in Figure FIGREF1  Others using artificially constrained action spaces also report an inability to pass through this bottleneck BIBREF7, BIBREF8  They pose a significant challenge for these methods because the agent does not see the correct action sequence to pass the bottleneck enough times  This is in part due to the fact that for that sequence to be reinforced, the agent needs to reach the next possible reward beyond the bottleneck 
More efficient exploration strategies are required to pass bottlenecks  Our contributions are two-fold  We first introduce a method that detects bottlenecks in text-games using the overall reward gained and the knowledge graph state  This method freezes the policy used to reach the bottleneck and restarts the training from there on out, additionally conducting a backtracking search to ensure that a sub-optimal policy has not been frozen  The second contribution explore how to leverage knowledge graphs to improve existing exploration algorithms for dealing with combinatorial action-spaces such as Go-Explore BIBREF9  We additionally present a comparative ablation study analyzing the performance of these methods on the popular text-game Zork1 
Exploration Methods
In this section, we describe methods to explore combinatorially sized action spaces such as text-games—focusing especially on methods that can deal with their inherent bottleneck structure  We first describe our method that explicitly attempts to detect bottlenecks and then describe how an exploration algorithm such as Go Explore BIBREF9 can leverage knowledge graphs 
KG-A2C-chained An example of a bottleneck can be seen in Figure FIGREF1  We extend the KG-A2C algorithm as follows  First, we detect bottlenecks as states where the agent is unable to progress any further  We set a patience parameter and if the agent has not seen a higher score in patience steps, the agent assumes it has been limited by a bottleneck  Second, when a bottleneck is found, we freeze the policy that gets the agent to the state with the highest score  The agent then begins training a new policy from that particular state 
Simply freezing the policy that led to the bottleneck, however, can potentially result in a policy one that is globally sub-optimal  We therefore employ a backtracking strategy that restarts exploration from each of the $n$ previous steps—searching for a more optimal policy that reaches that bottleneck  At each step, we keep track of a buffer of $n$ states and admissible actions that led up to that locally optimal state  We force the agent to explore from this state to attempt to drive it out of the local optima  If it is further unable to find itself out of this local optima, we refresh the training process again, but starting at the state immediately before the agent reaches the local optima  If this continues to fail, we continue to iterate through this buffer of seen states states up to that local optima until we either find a more optimal state or we run out of states to refresh from, in which we terminate the training algorithm 
KG-A2C-Explore Go-Explore BIBREF9 is an algorithm that is designed to keep track of sub-optimal and under-explored states in order to allow the agent to explore upon more optimal states that may be a result of sparse rewards  The Go-Explore algorithm consists of two phases, the first to continuously explore until a set of promising states and corresponding trajectories are found on the basis of total score, and the second to robustify this found policy against potential stochasticity in the game  Promising states are defined as those states when explored from will likely result in higher reward trajectories  Since the text games we are dealing with are mostly deterministic, with the exception of Zork in later stages, we only focus on using Phase 1 of the Go-Explore algorithm to find an optimal policy  BIBREF10 look at applying Go-Explore to text-games on a set of simpler games generated using the game generation framework TextWorld BIBREF1  Instead of training a policy network in parallel to generate actions used for exploration, they use a small set of “admissible actions”—actions guaranteed to change the world state at any given step during Phase 1—to explore and find high reward trajectories  This space of actions is relatively small (of the order of $10^2$ per step) and so finding high reward trajectories in larger action-spaces such as in Zork would be infeasible
Go-Explore maintains an archive of cells—defined as a set of states that map to a single representation—to keep track of promising states  BIBREF9 simply encodes each cell by keeping track of the agent's position and BIBREF10 use the textual observations encoded by recurrent neural network as a cell representation  We improve on this implementation by training the KG-A2C network in parallel, using the snapshot of the knowledge graph in conjunction with the game state to further encode the current state and use this as a cell representation  At each step, Go-Explore chooses a cell to explore at random (weighted by score to prefer more advanced cells)  The KG-A2C will run for a number of steps, starting with the knowledge graph state and the last seen state of the game from the cell  This will generate a trajectory for the agent while further training the KG-A2C at each iteration, creating a new representation for the knowledge graph as well as a new game state for the cell  After expanding a cell, Go-Explore will continue to sample cells by weight to continue expanding its known states  At the same time, KG-A2C will benefit from the heuristics of selecting preferred cells and be trained on promising states more often 
Evaluation
We compare our two exploration strategies to the following baselines and ablations:
KG-A2C This is the exact same method presented in BIBREF6 with no modifications 
A2C Represents the same approach as KG-A2C but with all the knowledge graph components removed  The state representation is text only encoded using recurrent networks 
A2C-chained Is a variation on KG-A2C-chained where we use our policy chaining approach with the A2C method to train the agent instead of KG-A2C 
A2C-Explore Uses A2C in addition to the exploration strategy seen in KG-A2C-Explore  The cell representations here are defined in terms of the recurrent network based encoding of the textual observation 
Figure FIGREF10 shows that agents utilizing knowledge-graphs in addition to either enhanced exploration method far outperform the baseline A2C and KG-A2C  KG-A2C-chained and KG-A2C-Explore both pass the bottleneck of a score of 40, whereas A2C-Explore gets to the bottleneck but cannot surpass it 
There are a couple of key insights that can be drawn from these results The first is that the knowledge graph appears to be critical; it is theorized to help with partial observability  However the knowledge graph representation isn't sufficient in that the knowledge graph representation without enhanced exploration methods cannot surpass the bottleneck  A2C-chained—which explores without a knowledge graph—fails to even outperform the baseline A2C  We hypothesize that this is due to the knowledge graph aiding implicitly in the sample efficiency of bottleneck detection and subsequent exploration  That is, exploring after backtracking from a potentially detected bottleneck is much more efficient in the knowledge graph based agent 
The Go-Explore based exploration algorithm sees less of a difference between agents  A2C-Explore converges more quickly, but to a lower reward trajectory that fails to pass the bottleneck, whereas KG-A2C-Explore takes longer to reach a similar reward but consistently makes it through the bottleneck  The knowledge graph cell representation appears to thus be a better indication of what a promising state is as opposed to just the textual observation 
Comparing the advanced exploration methods when using the knowledge graph, we see that both agents successfully pass the bottleneck corresponding to entering the cellar and lighting the lamp and reach comparable scores within a margin of error  KG-A2C-chained is significantly more sample efficient and converges faster  We can infer that chaining policies by explicitly detecting bottlenecks lets us pass it more quickly than attempting to find promising cell representations with Go-Explore  This form of chained exploration with backtracking is particularly suited to sequential decision making problems that can be represented as acyclic directed graphs as in Figure FIGREF1 
Appendix ::: Zork1
Zork1 is one of the first text-adventure games and heavily influences games released later in terms of narrative style and game structure  It is a dungeon crawler where the player must explore a vast world and collect a series of treasures  It was identified by BIBREF2 as a moonshot game and has been the subject of much work in leaning agents BIBREF12, BIBREF7, BIBREF11, BIBREF8  Rewards are given to the player when they collect treasures as well as when important intermediate milestones needed to further explore the world are passed  Figure FIGREF15 and Figure FIGREF1 show us a map of the world of Zork1 and the corresponding quest structure 
The bottleneck seen at a score of around 40 is when the player first enters the cellar on the right side of the map  The cellar is dark and you need to immediately light the lamp to see anything  Attempting to explore the cellar in the dark results in you being instantly killed by a monster known as a “grue” 
Appendix ::: Knowledge Graph Rules
We make no changes from the graph update rules used by BIBREF6  Candidate interactive objects are identified by performing part-of-speech tagging on the current observation, identifying singular and proper nouns as well as adjectives, and are then filtered by checking if they can be examined using the command $examine$ $OBJ$  Only the interactive objects not found in the inventory are linked to the node corresponding to the current room and the inventory items are linked to the “you” node  The only other rule applied uses the navigational actions performed by the agent to infer the relative positions of rooms, e g  $\langle kitchen,down,cellar \rangle $ when the agent performs $go$ $down$ when in the kitchen to move to the cellar 
Appendix ::: Hyperparameters
Hyperparameters used for our agents are given below  Patience and buffer size are used for the policy chaining method as described in Section SECREF2  Cell step size is a parameter used for Go-Explore and describes how many steps are taken when exploring in a given cell state  Base hyperparameters for KG-A2C are taken from BIBREF6 and the same parameters are used for A2C 

ground_truth: Reward of 11.8 for the A2C-chained model, 41.8 for the KG-A2C-chained model, 40 for A2C-Explore and 44 for KG-A2C-Explore.

KG-A2C-chained and KG-A2C-Explore both pass the bottleneck of a score of 40

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they report results only on English data?
Context 2: Introduction
Distributed word representations, commonly referred to as word embeddings BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , serve as elementary building blocks in the course of algorithm design for an expanding range of applications in natural language processing (NLP), including named entity recognition BIBREF4 , BIBREF5 , parsing BIBREF6 , sentiment analysis BIBREF7 , BIBREF8 , and word-sense disambiguation BIBREF9   Although the empirical utility of word embeddings as an unsupervised method for capturing the semantic or syntactic features of a certain word as it is used in a given lexical resource is well-established BIBREF10 , BIBREF11 , BIBREF12 , an understanding of what these features mean remains an open problem BIBREF13 , BIBREF14 and as such word embeddings mostly remain a black box  It is desirable to be able to develop insight into this black box and be able to interpret what it means, while retaining the utility of word embeddings as semantically-rich intermediate representations  Other than the intrinsic value of this insight, this would not only allow us to explain and understand how algorithms work BIBREF15 , but also set a ground that would facilitate the design of new algorithms in a more deliberate way 
Recent approaches to generating word embeddings (e g  BIBREF0 , BIBREF2 ) are rooted linguistically in the field of distributed semantics BIBREF16 , where words are taken to assume meaning mainly by their degree of interaction (or lack thereof) with other words in the lexicon BIBREF17 , BIBREF18   Under this paradigm, dense, continuous vector representations are learned in an unsupervised manner from a large corpus, using the word cooccurrence statistics directly or indirectly, and such an approach is shown to result in vector representations that mathematically capture various semantic and syntactic relations between words BIBREF0 , BIBREF2 , BIBREF3   However, the dense nature of the learned embeddings obfuscate the distinct concepts encoded in the different dimensions, which renders the resulting vectors virtually uninterpretable  The learned embeddings make sense only in relation to each other and their specific dimensions do not carry explicit information that can be interpreted  However, being able to interpret a word embedding would illuminate the semantic concepts implicitly represented along the various dimensions of the embedding, and reveal its hidden semantic structures 
In the literature, researchers tackled interpretability problem of the word embeddings using different approaches  Several researchers BIBREF19 , BIBREF20 , BIBREF21 proposed algorithms based on non-negative matrix factorization (NMF) applied to cooccurrence variant matrices  Other researchers suggested to obtain interpretable word vectors from existing uninterpretable word vectors by applying sparse coding BIBREF22 , BIBREF23 , by training a sparse auto-encoder to transform the embedding space BIBREF24 , by rotating the original embeddings BIBREF25 , BIBREF26 or by applying transformations based on external semantic datasets BIBREF27  
Although the above-mentioned approaches provide better interpretability that is measured using a particular method such as word intrusion test, usually the improved interpretability comes with a cost of performance in the benchmark tests such as word similarity or word analogy  One possible explanation for this performance decrease is that the proposed transformations from the original embedding space distort the underlying semantic structure constructed by the original embedding algorithm  Therefore, it can be claimed that a method that learns dense and interpretable word embeddings without inflicting any damage to the underlying semantic learning mechanism is the key to achieve both high performing and interpretable word embeddings 
Especially after the introduction of the word2vec algorithm by Mikolov BIBREF0 , BIBREF1 , there has been a growing interest in algorithms that generate improved word representations under some performance metric  Significant effort is spent on appropriately modifying the objective functions of the algorithms in order to incorporate knowledge from external resources, with the purpose of increasing the performance of the resulting word representations BIBREF28 , BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 , BIBREF37   Inspired by the line of work reported in these studies, we propose to use modified objective functions for a different purpose: learning more interpretable dense word embeddings  By doing this, we aim to incorporate semantic information from an external lexical resource into the word embedding so that the embedding dimensions are aligned along predefined concepts  This alignment is achieved by introducing a modification to the embedding learning process  In our proposed method, which is built on top of the GloVe algorithm BIBREF2 , the cost function for any one of the words of concept word-groups is modified by the introduction of an additive term to the cost function  Each embedding vector dimension is first associated with a concept  For a word belonging to any one of the word-groups representing these concepts, the modified cost term favors an increase for the value of this word's embedding vector dimension corresponding to the concept that the particular word belongs to  For words that do not belong to any one of the word-groups, the cost term is left untouched  Specifically, Roget's Thesaurus BIBREF38 , BIBREF39 is used to derive the concepts and concept word-groups to be used as the external lexical resource for our proposed method  We quantitatively demonstrate the increase in interpretability by using the measure given in BIBREF27 , BIBREF40 as well as demonstrating qualitative results  We also show that the semantic structure of the original embedding has not been harmed in the process since there is no performance loss with standard word-similarity or word-analogy tests 
The paper is organized as follows  In Section SECREF2 , we discuss previous studies related to our work under two main categories: interpretability of word embeddings and joint-learning frameworks where the objective function is modified  In Section SECREF3 , we present the problem framework and provide the formulation within the GloVe BIBREF2 algorithm setting  In Section SECREF4 where our approach is proposed, we motivate and develop a modification to the original objective function with the aim of increasing representation interpretability  In Section SECREF5 , experimental results are provided and the proposed method is quantitatively and qualitatively evaluated  Additionally, in Section SECREF5 , results demonstrating the extent to which the original semantic structure of the embedding space is affected are presented by using word-analogy and word-similarity tests  We conclude the paper in Section SECREF6  
Related Work
Methodologically, our work is related to prior studies that aim to obtain “improved” word embeddings using external lexical resources, under some performance metric  Previous work in this area can be divided into two main categories: works that i) modify the word embedding learning algorithm to incorporate lexical information, ii) operate on pre-trained embeddings with a post-processing step 
Among works that follow the first approach, BIBREF28 extend the Skip-Gram model by incorporating the word similarity relations extracted from the Paraphrase Database (PPDB) and WordNet BIBREF29 , into the Skip-Gram predictive model as an additional cost term  In BIBREF30 , the authors extend the CBOW model by considering two types of semantic information, termed relational and categorical, to be incorporated into the embeddings during training  For the former type of semantic information, the authors propose the learning of explicit vectors for the different relations extracted from a semantic lexicon such that the word pairs that satisfy the same relation are distributed more homogeneously  For the latter, the authors modify the learning objective such that some weighted average distance is minimized for words under the same semantic category  In BIBREF31 , the authors represent the synonymy and hypernymy-hyponymy relations in terms of inequality constraints, where the pairwise similarity rankings over word triplets are forced to follow an order extracted from a lexical resource  Following their extraction from WordNet, the authors impose these constraints in the form of an additive cost term to the Skip-Gram formulation  Finally, BIBREF32 builds on top of the GloVe algorithm by introducing a regularization term to the objective function that encourages the vector representations of similar words as dictated by WordNet to be similar as well 
Turning our attention to the post-processing approach for enriching word embeddings with external lexical knowledge, BIBREF33 has introduced the retrofitting algorithm that acts on pre-trained embeddings such as Skip-Gram or GloVe  The authors propose an objective function that aims to balance out the semantic information captured in the pre-trained embeddings with the constraints derived from lexical resources such as WordNet, PPDB and FrameNet  One of the models proposed in BIBREF34 extends the retrofitting approach to incorporate the word sense information from WordNet  Similarly, BIBREF35 creates multi-sense embeddings by gathering the word sense information from a lexical resource and learning to decompose the pre-trained embeddings into a convex combination of sense embeddings  In BIBREF36 , the authors focus on improving word embeddings for capturing word similarity, as opposed to mere relatedness  To this end, they introduce the counter-fitting technique which acts on the input word vectors such that synonymous words are attracted to one another whereas antonymous words are repelled, where the synonymy-antonymy relations are extracted from a lexical resource  More recently, the ATTRACT-REPEL algorithm proposed by BIBREF37 improves on counter-fitting by a formulation which imparts the word vectors with external lexical information in mini-batches 
Most of the studies discussed above ( BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF36 , BIBREF37 ) report performance improvements in benchmark tests such as word similarity or word analogy, while BIBREF29 uses a different analysis method (mean reciprocal rank)  In sum, the literature is rich with studies aiming to obtain word embeddings that perform better under specific performance metrics  However, less attention has been directed to the issue of interpretability of the word embeddings  In the literature, the problem of interpretability has been tackled using different approaches  BIBREF19 proposed non-negative matrix factorization (NMF) for learning sparse, interpretable word vectors from co-occurrence variant matrices where the resulting vector space is called non-negative sparse embeddigns (NNSE)  However, since NMF methods require maintaining a global matrix for learning, they suffer from memory and scale issue  This problem has been addressed in BIBREF20 where an online method of learning interpretable word embeddings from corpora using a modified version of skip-gram model BIBREF0 is proposed  As a different approach, BIBREF21 combined text-based similarity information among words with brain activity based similarity information to improve interpretability using joint non-negative sparse embedding (JNNSE) 
A common alternative approach for learning interpretable embeddings is to learn transformations that map pre-trained state-of-the-art embeddings to new interpretable semantic spaces  To obtain sparse, higher dimensional and more interpretable vector spaces, BIBREF22 and BIBREF23 use sparse coding on conventional dense word embeddings  However, these methods learn the projection vectors that are used for the transformation from the word embeddings without supervision  For this reason, labels describing the corresponding semantic categories cannot be provided  An alternative approach was proposed in BIBREF25 , where orthogonal transformations were utilized to increase interpretability while preserving the performance of the underlying embedding  However, BIBREF25 has also shown that total interpretability of an embedding is kept constant under any orthogonal transformation and it can only be redistributed across the dimensions  Rotation algorithms based on exploratory factor analysis (EFA) to preserve the performance of the original word embeddings while improving their interpretability was proposed in BIBREF26   BIBREF24 proposed to deploy a sparse auto-encoder using pre-trained dense word embeddings to improve interpretability  More detailed investigation of semantic structure and interpretability of word embeddings can be found in BIBREF27 , where a metric was proposed to quantitatively measure the degree of interpretability already present in the embedding vector spaces 
Previous works on interpretability mentioned above, except BIBREF21 , BIBREF27 and our proposed method, do not need external resources, utilization of which has both advantages and disadvantages  Methods that do not use external resources require fewer resources but they also lack the aid of information extracted from these resources 
Problem Description
For the task of unsupervised word embedding extraction, we operate on a discrete collection of lexical units (words) INLINEFORM0 that is part of an input corpus INLINEFORM1 , with number of tokens INLINEFORM2 , sourced from a vocabulary INLINEFORM3 of size INLINEFORM4   In the setting of distributional semantics, the objective of a word embedding algorithm is to maximize some aggregate utility over the entire corpus so that some measure of “closeness” is maximized for pairs of vector representations INLINEFORM14 for words which, on the average, appear in proximity to one another  In the GloVe algorithm BIBREF2 , which we base our improvements upon, the following objective function is considered: DISPLAYFORM0
In ( EQREF6 ), INLINEFORM0 and INLINEFORM1 stand for word and context vector representations, respectively, for words INLINEFORM2 and INLINEFORM3 , while INLINEFORM4 represents the (possibly weighted) cooccurrence count for the word pair INLINEFORM5   Intuitively, ( EQREF6 ) represents the requirement that if some word INLINEFORM6 occurs often enough in the context (or vicinity) of another word INLINEFORM7 , then the corresponding word representations should have a large enough inner product in keeping with their large INLINEFORM8 value, up to some bias terms INLINEFORM9 ; and vice versa  INLINEFORM10 in ( EQREF6 ) is used as a discounting factor that prohibits rare cooccurrences from disproportionately influencing the resulting embeddings 
The objective ( EQREF6 ) is minimized using stochastic gradient descent by iterating over the matrix of cooccurrence records INLINEFORM0   In the GloVe algorithm, for a given word INLINEFORM1 , the final word representation is taken to be the average of the two intermediate vector representations obtained from ( EQREF6 ); i e, INLINEFORM2   In the next section, we detail the enhancements made to ( EQREF6 ) for the purposes of enhanced interpretability, using the aforementioned framework as our basis 
Imparting Interpretability
Our approach falls into a joint-learning framework where the distributional information extracted from the corpus is allowed to fuse with the external lexicon-based information  Word-groups extracted from Roget's Thesaurus are directly mapped to individual dimensions of word embeddings  Specifically, the vector representations of words that belong to a particular group are encouraged to have deliberately increased values in a particular dimension that corresponds to the word-group under consideration  This can be achieved by modifying the objective function of the embedding algorithm to partially influence vector representation distributions across their dimensions over an input vocabulary  To do this, we propose the following modification to the GloVe objective in ( EQREF6 ): rCl J = i,j=1V f(Xij)[ (wiTwj + bi + bj -Xij)2
+ k(l=1D INLINEFORM0 iFl g(wi,l) + l=1D INLINEFORM1 j Fl g(wj,l) ) ]  In ( SECREF4 ), INLINEFORM2 denotes the indices for the elements of the INLINEFORM3 th concept word-group which we wish to assign in the vector dimension INLINEFORM4   The objective ( SECREF4 ) is designed as a mixture of two individual cost terms: the original GloVe cost term along with a second term that encourages embedding vectors of a given concept word-group to achieve deliberately increased values along an associated dimension INLINEFORM5   The relative weight of the second term is controlled by the parameter INLINEFORM6   The simultaneous minimization of both objectives ensures that words that are similar to, but not included in, one of these concept word-groups are also “nudged” towards the associated dimension INLINEFORM7   The trained word vectors are thus encouraged to form a distribution where the individual vector dimensions align with certain semantic concepts represented by a collection of concept word-groups, one assigned to each vector dimension  To facilitate this behaviour, ( SECREF4 ) introduces a monotone decreasing function INLINEFORM8 defined as INLINEFORM9
which serves to increase the total cost incurred if the value of the INLINEFORM0 th dimension for the two vector representations INLINEFORM1 and INLINEFORM2 for a concept word INLINEFORM3 with INLINEFORM4 fails to be large enough  INLINEFORM5 is also shown in Fig  FIGREF7  
The objective ( SECREF4 ) is minimized using stochastic gradient descent over the cooccurrence records INLINEFORM0   Intuitively, the terms added to ( SECREF4 ) in comparison with ( EQREF6 ) introduce the effect of selectively applying a positive step-type input to the original descent updates of ( EQREF6 ) for concept words along their respective vector dimensions, which influences the dimension value in the positive direction  The parameter INLINEFORM1 in ( SECREF4 ) allows for the adjustment of the magnitude of this influence as needed 
In the next section, we demonstrate the feasibility of this approach by experiments with an example collection of concept word-groups extracted from Roget's Thesaurus 
Experiments and Results
We first identified 300 concepts, one for each dimension of the 300-dimensional vector representation, by employing Roget's Thesaurus  This thesaurus follows a tree structure which starts with a Root node that contains all the words and phrases in the thesaurus  The root node is successively split into Classes and Sections, which are then (optionally) split into Subsections of various depths, finally ending in Categories, which constitute the smallest unit of word/phrase collections in the structure  The actual words and phrases descend from these Categories, and make up the leaves of the tree structure  We note that a given word typically appears in multiple categories corresponding to the different senses of the word  We constructed concept word-groups from Roget's Thesaurus as follows: We first filtered out the multi-word phrases and the relatively obscure terms from the thesaurus  The obscure terms were identified by checking them against a vocabulary extracted from Wikipedia  We then obtained 300 word-groups as the result of a partitioning operation applied to the subtree that ends with categories as its leaves  The partition boundaries, hence the resulting word-groups, can be chosen in many different ways  In our proposed approach, we have chosen to determine this partitioning by traversing this tree structure from the root node in breadth-first order, and by employing a parameter INLINEFORM0 for the maximum size of a node  Here, the size of a node is defined as the number of unique words that ever-descend from that node  During the traversal, if the size of a given node is less than this threshold, we designate the words that ultimately descend from that node as a concept word-group  Otherwise, if the node has children, we discard the node, and queue up all its children for further consideration  If this node does not have any children, on the other hand, the node is truncated to INLINEFORM1 elements with the highest frequency-ranks, and the resulting words are designated as a concept word-group  We note that the choice of INLINEFORM2 greatly affects the resulting collection of word-groups: Excessively large values result in few word-groups that greatly overlap with one another, while overly small values result in numerous tiny word-groups that fail to adequately represent a concept  We experimentally determined that a INLINEFORM3 value of 452 results in the most healthy number of relatively large word-groups (113 groups with size INLINEFORM4 100), while yielding a preferably small overlap amongst the resulting word-groups (with average overlap size not exceeding 3 words)  A total of 566 word-groups were thus obtained  259 smallest word-groups (with size INLINEFORM5 38) were discarded to bring down the number of word-groups to 307  Out of these, 7 groups with the lowest median frequency-rank were further discarded, which yields the final 300 concept word-groups used in the experiments  We present some of the resulting word-groups in Table TABREF9  
By using the concept word-groups, we have trained the GloVe algorithm with the proposed modification given in Section SECREF4 on a snapshot of English Wikipedia measuring 8GB in size, with the stop-words filtered out  Using the parameters given in Table TABREF10 , this resulted in a vocabulary size of 287,847  For the weighting parameter in Eq  SECREF4 , we used a value of INLINEFORM0   The algorithm was trained over 20 iterations  The GloVe algorithm without any modifications was also trained as a baseline with the same parameters  In addition to the original GloVe algorithm, we compare our proposed method with previous studies that aim to obtain interpretable word vectors  We train the improved projected gradient model proposed in BIBREF20 to obtain word vectors (called OIWE-IPG) using the same corpus we use to train GloVe and our proposed method  Using the methods proposed in BIBREF23 , BIBREF26 , BIBREF24 on our baseline GloVe embeddings, we obtain SOV, SPINE and Parsimax (orthogonal) word representations, respectively  We train all the models with the proposed parameters  However, in BIBREF26 , the authors show results for a relatively small vocabulary of 15,000 words  When we trained their model on our baseline GloVe embeddings with a large vocabulary of size 287,847, the resulting vectors performed significantly poor on word similarity tasks compared to the results presented in their paper  In addition, Parsimax (orthogonal) word vectors obtained using method in BIBREF26 are nearly identical to the baseline vectors (i e  learned orthogonal transformation matrix is very close to identity)  Therefore, Parsimax (orthogonal) yields almost same results with baseline vectors in all evaluations  We evaluate the interpretability of the resulting embeddings qualitatively and quantitatively  We also test the performance of the embeddings on word similarity and word analogy tests 
In our experiments, vocabulary size is close to 300,000 while only 16,242 unique words of the vocabulary are present in the concept groups  Furthermore, only dimensions that correspond to the concept group of the word will be updated due to the additional cost term  Given that these concept words can belong to multiple concept groups (2 on average), only 33,319 parameters are updated  There are 90 million individual parameters present for the 300,000 word vectors of size 300  Of these parameters, only approximately 33,000 are updated by the additional cost term 
Qualitative Evaluation for Interpretability
In Fig  FIGREF13 , we demonstrate the particular way in which the proposed algorithm ( SECREF4 ) influences the vector representation distributions  Specifically, we consider, for illustration, the 32nd dimension values for the original GloVe algorithm and our modified version, restricting the plots to the top-1000 words with respect to their frequency ranks for clarity of presentation  In Fig  FIGREF13 , the words in the horizontal axis are sorted in descending order with respect to the values at the 32nd dimension of their word embedding vectors coming from the original GloVe algorithm  The dimension values are denoted with blue and red/green markers for the original and the proposed algorithms, respectively  Additionally, the top-50 words that achieve the greatest 32nd dimension values among the considered 1000 words are emphasized with enlarged markers, along with text annotations  In the presented simulation of the proposed algorithm, the 32nd dimension values are encoded with the concept JUDGMENT, which is reflected as an increase in the dimension values for words such as committee, academy, and article  We note that these words (red) are not part of the pre-determined word-group for the concept JUDGMENT, in contrast to words such as award, review and account (green) which are  This implies that the increase in the corresponding dimension values seen for these words is attributable to the joint effect of the first term in ( SECREF4 ) which is inherited from the original GloVe algorithm, in conjunction with the remaining terms in the proposed objective expression ( SECREF4 )  This experiment illustrates that the proposed algorithm is able to impart the concept of JUDGMENT on its designated vector dimension above and beyond the supplied list of words belonging to the concept word-group for that dimension  We also present the list of words with the greatest dimension value for the dimensions 11, 13, 16, 31, 36, 39, 41, 43 and 79 in Table TABREF11   These dimensions are aligned/imparted with the concepts that are given in the column headers  In Table TABREF11 , the words that are highlighted with green denote the words that exist in the corresponding word-group obtained from Roget's Thesaurus (and are thus explicitly forced to achieve increased dimension values), while the red words denote the words that achieve increased dimension values by virtue of their cooccurrence statistics with the thesaurus-based words (indirectly, without being explicitly forced)  This again illustrates that a semantic concept can indeed be coded to a vector dimension provided that a sensible lexical resource is used to guide semantically related words to the desired vector dimension via the proposed objective function in ( SECREF4 )  Even the words that do not appear in, but are semantically related to, the word-groups that we formed using Roget's Thesaurus, are indirectly affected by the proposed algorithm  They also reflect the associated concepts at their respective dimensions even though the objective functions for their particular vectors are not modified  This point cannot be overemphasized  Although the word-groups extracted from Roget's Thesaurus impose a degree of supervision to the process, the fact that the remaining words in the entire vocabulary are also indirectly affected makes the proposed method a semi-supervised approach that can handle words that are not in these chosen word-groups  A qualitative example of this result can be seen in the last column of Table TABREF11   It is interesting to note the appearance of words such as guerilla, insurgency, mujahideen, Wehrmacht and Luftwaffe in addition to the more obvious and straightforward army, soldiers and troops, all of which are not present in the associated word-group WARFARE 
Most of the dimensions we investigated exhibit similar behaviour to the ones presented in Table TABREF11   Thus generally speaking, we can say that the entries in Table TABREF11 are representative of the great majority  However, we have also specifically looked for dimensions that make less sense and determined a few such dimensions which are relatively less satisfactory  These less satisfactory examples are given in Table TABREF14   These examples are also interesting in that they shed insight into the limitations posed by polysemy and existence of very rare outlier words 
Quantitative Evaluation for Interpretability
One of the main goals of this study is to improve the interpretability of dense word embeddings by aligning the dimensions with predefined concepts from a suitable lexicon  A quantitative measure is required to reliably evaluate the achieved improvement  One of the methods proposed to measure the interpretability is the word intrusion test BIBREF41   But, this method is expensive to apply since it requires evaluations from multiple human evaluators for each embedding dimension  In this study, we use a semantic category-based approach based on the method and category dataset (SEMCAT) introduced in BIBREF27 to quantify interpretability  Specifically, we apply a modified version of the approach presented in BIBREF40 in order to consider possible sub-groupings within the categories in SEMCAT  Interpretability scores are calculated using Interpretability Score (IS) as given below:
DISPLAYFORM0
In ( EQREF17 ), INLINEFORM0 and INLINEFORM1 represents the interpretability scores in the positive and negative directions of the INLINEFORM2 dimension ( INLINEFORM3 , INLINEFORM4 number of dimensions in the embedding space) of word embedding space for the INLINEFORM5 category ( INLINEFORM6 , INLINEFORM7 is number of categories in SEMCAT, INLINEFORM8 ) in SEMCAT respectively  INLINEFORM9 is the set of words in the INLINEFORM10 category in SEMCAT and INLINEFORM11 is the number of words in INLINEFORM12   INLINEFORM13 corresponds to the minimum number of words required to construct a semantic category (i e  represent a concept)  INLINEFORM14 represents the set of INLINEFORM15 words that have the highest ( INLINEFORM16 ) and lowest ( INLINEFORM17 ) values in INLINEFORM18 dimension of the embedding space  INLINEFORM19 is the intersection operator and INLINEFORM20 is the cardinality operator (number of elements) for the intersecting set  In ( EQREF17 ), INLINEFORM21 gives the interpretability score for the INLINEFORM22 dimension and INLINEFORM23 gives the average interpretability score of the embedding space 
Fig  FIGREF18 presents the measured average interpretability scores across dimensions for original GloVe embeddings, for the proposed method and for the other four methods we compare, along with a randomly generated embedding  Results are calculated for the parameters INLINEFORM0 and INLINEFORM1   Our proposed method significantly improves the interpretability for all INLINEFORM2 compared to the original GloVe approach  Our proposed method is second to only SPINE in increasing interpretability  However, as we will experimentally demonstrate in the next subsection, in doing this, SPINE almost entirely destroys the underlying semantic structure of the word embeddings, which is the primary function of a word embedding 
The proposed method and interpretability measurements are both based on utilizing concepts represented by word-groups  Therefore it is expected that there will be higher interpretability scores for some of the dimensions for which the imparted concepts are also contained in SEMCAT  However, by design, word groups that they use are formed by using different sources and are independent  Interpretability measurements use SEMCAT while our proposed method utilizes Roget's Thesaurus 
Intrinsic Evaluation of the Embeddings
It is necessary to show that the semantic structure of the original embedding has not been damaged or distorted as a result of aligning the dimensions with given concepts, and that there is no substantial sacrifice involved from the performance that can be obtained with the original GloVe  To check this, we evaluate performances of the proposed embeddings on word similarity BIBREF42 and word analogy BIBREF0 tests  We compare the results with the original embeddings and the three alternatives excluding Parsimax BIBREF26 since orthogonal transformations will not affect the performance of the original embeddings on these tests 
Word similarity test measures the correlation between word similarity scores obtained from human evaluation (i e  true similarities) and from word embeddings (usually using cosine similarity)  In other words, this test quantifies how well the embedding space reflects human judgements in terms of similarities between different words  The correlation scores for 13 different similarity test sets are reported in Table TABREF20   We observe that, let alone a reduction in performance, the obtained scores indicate an almost uniform improvement in the correlation values for the proposed algorithm, outperforming all the alternatives in almost all test sets  Categories from Roget's thesaurus are groupings of words that are similar in some sense which the original embedding algorithm may fail to capture  These test results signify that the semantic information injected into the algorithm by the additional cost term is significant enough to result in a measurable improvement  It should also be noted that scores obtained by SPINE is unacceptably low on almost all tests indicating that it has achieved its interpretability performance at the cost of losing its semantic functions 
Word analogy test is introduced in BIBREF1 and looks for the answers of the questions that are in the form of "X is to Y, what Z is to ?" by applying simple arithmetic operations to vectors of words X, Y and Z  We present precision scores for the word analogy tests in Table TABREF21   It can be seen that the alternative approaches that aim to improve interpretability, have poor performance on the word analogy tests  However, our proposed method has comparable performance with the original GloVe embeddings  Our method outperforms GloVe in semantic analogy test set and in overall results, while GloVe performs slightly better in syntactic test set  This comparable performance is mainly due to the cost function of our proposed method that includes the original objective of the GloVe 
To investigate the effect of the additional cost term on the performance improvement in the semantic analogy test, we present Table TABREF22   In particular, we present results for the cases where i) all questions in the dataset are considered, ii) only the questions that contains at least one concept word are considered, iii) only the questions that consist entirely of concept words are considered  We note specifically that for the last case, only a subset of the questions under the semantic category family txt ended up being included  We observe that for all three scenarios, our proposed algorithm results in an improvement in the precision scores  However, the greatest performance increase is seen for the last scenario, which underscores the extent to which the semantic features captured by embeddings can be improved with a reasonable selection of the lexical resource from which the concept word-groups were derived 
Conclusion
We presented a novel approach to impart interpretability into word embeddings  We achieved this by encouraging different dimensions of the vector representation to align with predefined concepts, through the addition of an additional cost term in the optimization objective of the GloVe algorithm that favors a selective increase for a pre-specified input of concept words along each dimension 
We demonstrated the efficacy of this approach by applying qualitative and quantitative evaluations for interpretability  We also showed via standard word-analogy and word-similarity tests that the semantic coherence of the original vector space is preserved, even slightly improved  We have also performed and reported quantitative comparisons with several other methods for both interpretabilty increase and preservation of semantic coherence  Upon inspection of Fig  FIGREF18 and Tables TABREF20 , TABREF21 , and TABREF22 altogether, it should be noted that our proposed method achieves both of the objectives simultaneously, increased interpretability and preservation of the intrinsic semantic structure 
An important point was that, while it is expected for words that are already included in the concept word-groups to be aligned together since their dimensions are directly updated with the proposed cost term, it was also observed that words not in these groups also aligned in a meaningful manner without any direct modification to their cost function  This indicates that the cost term we added works productively with the original cost function of GloVe to handle words that are not included in the original concept word-groups, but are semantically related to those word-groups  The underlying mechanism can be explained as follows  While the outside lexical resource we introduce contains a relatively small number of words compared to the total number of words, these words and the categories they represent have been carefully chosen and in a sense, "densely span" all the words in the language  By saying "span", we mean they cover most of the concepts and ideas in the language without leaving too many uncovered areas  With "densely" we mean all areas are covered with sufficient strength  In other words, this subset of words is able to constitute a sufficiently strong skeleton, or scaffold  Now remember that GloVe works to align or bring closer related groups of words, which will include words from the lexical source  So the joint action of aligning the words with the predefined categories (introduced by us) and aligning related words (handled by GloVe) allows words not in the lexical groups to also be aligned meaningfully  We may say that the non-included words are "pulled along" with the included words by virtue of the "strings" or "glue" that is provided by GloVe  In numbers, the desired effect is achieved by manipulating less than only 0 05% of parameters of the entire word vectors  Thus, while there is a degree of supervision coming from the external lexical resource, the rest of the vocabulary is also aligned indirectly in an unsupervised way  This may be the reason why, unlike earlier proposed approaches, our method is able to achieve increasing interpretability without destroying underlying semantic structure, and consequently without sacrificing performance in benchmark tests 
Upon inspecting the 2nd column of Table TABREF14 , where qualitative results for concept TASTE are presented, another insight regarding the learning mechanism of our proposed approach can be made  Here it seems understandable that our proposed approach, along with GloVe, brought together the words taste and polish, and then the words Polish and, for instance, Warsaw are brought together by GloVe  These examples are interesting in that they shed insight into how GloVe works and the limitations posed by polysemy  It should be underlined that the present approach is not totally incapable of handling polysemy, but cannot do so perfectly  Since related words are being clustered, sufficiently well-connected words that do not meaningfully belong along with others will be appropriately "pulled away" from that group by several words, against the less effective, inappropriate pull of a particular word  Even though polish with lowercase "p" belongs where it is, it is attracting Warsaw to itself through polysemy and this is not meaningful  Perhaps because Warsaw is not a sufficiently well-connected word, it ends being dragged along, although words with greater connectedness to a concept group might have better resisted such inappropriate attractions 
In this study, we used the GloVe algorithm as the underlying dense word embedding scheme to demonstrate our approach  However, we stress that it is possible for our approach to be extended to other word embedding algorithms which have a learning routine consisting of iterations over cooccurrence records, by making suitable adjustments in the objective function  Since word2vec model is also based on the coocurrences of words in a sliding window through a large corpus, we expect that our approach can also be applied to word2vec after making suitable adjustments, which can be considered as an immediate future work for our approach  Although the semantic concepts are encoded in only one direction (positive) within the embedding dimensions, it might be beneficial to pursue future work that also encodes opposite concepts, such as good and bad, in two opposite directions of the same dimension 
The proposed methodology can also be helpful in computational cross-lingual studies, where the similarities are explored across the vector spaces of different languages BIBREF43 , BIBREF44  
Context 3: Introduction
Data annotation is a major bottleneck for the application of supervised learning approaches to many problems  As a result, unsupervised methods that learn directly from unlabeled data are increasingly important  For tasks related to unsupervised syntactic analysis, discrete generative models have dominated in recent years – for example, for both part-of-speech (POS) induction BIBREF0 , BIBREF1 and unsupervised dependency parsing BIBREF2 , BIBREF3 , BIBREF4   While similar models have had success on a range of unsupervised tasks, they have mostly ignored the apparent utility of continuous word representations evident from supervised NLP applications BIBREF5 , BIBREF6   In this work, we focus on leveraging and explicitly representing continuous word embeddings within unsupervised models of syntactic structure 
Pre-trained word embeddings from massive unlabeled corpora offer a compact way of injecting a prior notion of word similarity into models that would otherwise treat words as discrete, isolated categories  However, the specific properties of language captured by any particular embedding scheme can be difficult to control, and, further, may not be ideally suited to the task at hand  For example, pre-trained skip-gram embeddings BIBREF7 with small context window size are found to capture the syntactic properties of language well BIBREF8 , BIBREF9   However, if our goal is to separate syntactic categories, this embedding space is not ideal – POS categories correspond to overlapping interspersed regions in the embedding space, evident in Figure SECREF4  
In our approach, we propose to learn a new latent embedding space as a projection of pre-trained embeddings (depicted in Figure SECREF5 ), while jointly learning latent syntactic structure – for example, POS categories or syntactic dependencies  To this end, we introduce a new generative model (shown in Figure FIGREF6 ) that first generates a latent syntactic representation (e g  a dependency parse) from a discrete structured prior (which we also call the “syntax model”), then, conditioned on this representation, generates a sequence of latent embedding random variables corresponding to each word, and finally produces the observed (pre-trained) word embeddings by projecting these latent vectors through a parameterized non-linear function  The latent embeddings can be jointly learned with the structured syntax model in a completely unsupervised fashion 
By choosing an invertible neural network as our non-linear projector, and then parameterizing our model in terms of the projection's inverse, we are able to derive tractable exact inference and marginal likelihood computation procedures so long as inference is tractable in the underlying syntax model  In sec:learn-with-inv we show that this derivation corresponds to an alternate view of our approach whereby we jointly learn a mapping of observed word embeddings to a new embedding space that is more suitable for the syntax model, but include an additional Jacobian regularization term to prevent information loss 
Recent work has sought to take advantage of word embeddings in unsupervised generative models with alternate approaches BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12   BIBREF9 build an HMM with Gaussian emissions on observed word embeddings, but they do not attempt to learn new embeddings  BIBREF10 , BIBREF11 , and BIBREF12 extend HMM or dependency model with valence (DMV) BIBREF2 with multinomials that use word (or tag) embeddings in their parameterization  However, they do not represent the embeddings as latent variables 
In experiments, we instantiate our approach using both a Markov-structured syntax model and a tree-structured syntax model – specifically, the DMV  We evaluate on two tasks: part-of-speech (POS) induction and unsupervised dependency parsing without gold POS tags  Experimental results on the Penn Treebank BIBREF13 demonstrate that our approach improves the basic HMM and DMV by a large margin, leading to the state-of-the-art results on POS induction, and state-of-the-art results on unsupervised dependency parsing in the difficult training scenario where neither gold POS annotation nor punctuation-based constraints are available 
Model
As an illustrative example, we first present a baseline model for Markov syntactic structure (POS induction) that treats a sequence of pre-trained word embeddings as observations  Then, we propose our novel approach, again using Markov structure, that introduces latent word embedding variables and a neural projector  Lastly, we extend our approach to more general syntactic structures 
Example: Gaussian HMM
We start by describing the Gaussian hidden Markov model introduced by BIBREF9 , which is a locally normalized model with multinomial transitions and Gaussian emissions  Given a sentence of length INLINEFORM0 , we denote the latent POS tags as INLINEFORM1 , observed (pre-trained) word embeddings as INLINEFORM2 , transition parameters as INLINEFORM3 , and Gaussian emission parameters as INLINEFORM4   The joint distribution of data and latent variables factors as:
DISPLAYFORM0
where INLINEFORM0 is the multinomial transition probability and INLINEFORM1 is the multivariate Gaussian emission probability 
While the observed word embeddings do inform this model with a notion of word similarity – lacking in the basic multinomial HMM – the Gaussian emissions may not be sufficiently flexible to separate some syntactic categories in the complex pre-trained embedding space – for example the skip-gram embedding space as visualized in Figure SECREF4 where different POS categories overlap  Next we introduce a new approach that adds flexibility to the emission distribution by incorporating new latent embedding variables 
Markov Structure with Neural Projector
To flexibly model observed embeddings and yield a new representation space that is more suitable for the syntax model, we propose to cascade a neural network as a projection function, deterministically transforming the simple space defined by the Gaussian HMM to the observed embedding space  We denote the latent embedding of the INLINEFORM0 word in a sentence as INLINEFORM1 , and the neural projection function as INLINEFORM2 , parameterized by INLINEFORM3   In the case of sequential Markov structure, our new model corresponds to the following generative process:
For each time step INLINEFORM0 ,
[noitemsep, leftmargin=*]
Draw the latent state INLINEFORM0
Draw the latent embedding INLINEFORM0
Deterministically produce embedding
INLINEFORM0
The graphical model is depicted in Figure FIGREF6   The deterministic projection can also be viewed as sampling each observation from a point mass at INLINEFORM0   The joint distribution of our model is: DISPLAYFORM0
where INLINEFORM0 is a conditional Gaussian distribution, and INLINEFORM1 is the Dirac delta function centered at INLINEFORM2 : DISPLAYFORM0
General Structure with Neural Projector
Our approach can be applied to a broad family of structured syntax models  We denote latent embedding variables as INLINEFORM0 , discrete latent variables in the syntax model as INLINEFORM1 ( INLINEFORM2 ), where INLINEFORM3 are conditioned to generate INLINEFORM4   The joint probability of our model factors as:
DISPLAYFORM0
where INLINEFORM0 represents the probability of the syntax model, and can encode any syntactic structure – though, its factorization structure will determine whether inference is tractable in our full model  As shown in Figure FIGREF6 , we focus on two syntax models for syntactic analysis in this paper  The first is Markov-structured, which we use for POS induction, and the second is DMV-structured, which we use to learn dependency parses without supervision 
The marginal data likelihood of our model is: DISPLAYFORM0
While the discrete variables INLINEFORM0 can be marginalized out with dynamic program in many cases, it is generally intractable to marginalize out the latent continuous variables, INLINEFORM1 , for an arbitrary projection INLINEFORM2 in Eq  ( EQREF17 ), which means inference and learning may be difficult  In sec:opt, we address this issue by constraining INLINEFORM3 to be invertible, and show that this constraint enables tractable exact inference and marginal likelihood computation 
Learning & Inference
In this section, we introduce an invertibility condition for our neural projector to tackle the optimization challenge  Specifically, we constrain our neural projector with two requirements: (1) INLINEFORM0 and (2) INLINEFORM1 exists  Invertible transformations have been explored before in independent components analysis BIBREF14 , gaussianization BIBREF15 , and deep density models BIBREF16 , BIBREF17 , BIBREF18 , for unstructured data  Here, we generalize this style of approach to structured learning, and augment it with discrete latent variables ( INLINEFORM2 )  Under the invertibility condition, we derive a learning algorithm and give another view of our approach revealed by the objective function  Then, we present the architecture of a neural projector we use in experiments: a volume-preserving invertible neural network proposed by BIBREF16 for independent components estimation 
Learning with Invertibility
For ease of exposition, we explain the learning algorithm in terms of Markov structure without loss of generality  As shown in Eq  ( EQREF17 ), the optimization challenge in our approach comes from the intractability of the marginalized emission factor INLINEFORM0   If we can marginalize out INLINEFORM1 and compute INLINEFORM2 , then the posterior and marginal likelihood of our Markov-structured model can be computed with the forward-backward algorithm  We can apply Eq  ( EQREF14 ) and obtain : INLINEFORM3
By using the change of variable rule to the integration, which allows the integration variable INLINEFORM0 to be replaced by INLINEFORM1 , the marginal emission factor can be computed in closed-form when the invertibility condition is satisfied: DISPLAYFORM0
where INLINEFORM0 is a conditional Gaussian distribution, INLINEFORM1 is the Jacobian matrix of function INLINEFORM2 at INLINEFORM3 , and INLINEFORM4 represents the absolute value of its determinant  This Jacobian term is nonzero and differentiable if and only if INLINEFORM5 exists 
Eq  ( EQREF19 ) shows that we can directly calculate the marginal emission distribution INLINEFORM0   Denote the marginal data likelihood of Gaussian HMM as INLINEFORM1 , then the log marginal data likelihood of our model can be directly written as: DISPLAYFORM0
where INLINEFORM0 represents the new sequence of embeddings after applying INLINEFORM1 to each INLINEFORM2   Eq  ( EQREF20 ) shows that the training objective of our model is simply the Gaussian HMM log likelihood with an additional Jacobian regularization term  From this view, our approach can be seen as equivalent to reversely projecting the data through INLINEFORM3 to another manifold INLINEFORM4 that is directly modeled by the Gaussian HMM, with a regularization term  Intuitively, we optimize the reverse projection INLINEFORM5 to modify the INLINEFORM6 space, making it more appropriate for the syntax model  The Jacobian regularization term accounts for the volume expansion or contraction behavior of the projection  Maximizing it can be thought of as preventing information loss  In the extreme case, the Jacobian determinant is equal to zero, which means the projection is non-invertible and thus information is being lost through the projection  Such “information preserving” regularization is crucial during optimization, otherwise the trivial solution of always projecting data to the same single point to maximize likelihood is viable 
More generally, for an arbitrary syntax model the data likelihood of our approach is: DISPLAYFORM0
If the syntax model itself allows for tractable inference and marginal likelihood computation, the same dynamic program can be used to marginalize out INLINEFORM0   Therefore, our joint model inherits the tractability of the underlying syntax model 
Invertible Volume-Preserving Neural Net
For the projection we can use an arbitrary invertible function, and given the representational power of neural networks they seem a natural choice  However, calculating the inverse and Jacobian of an arbitrary neural network can be difficult, as it requires that all component functions be invertible and also requires storage of large Jacobian matrices, which is memory intensive  To address this issue, several recent papers propose specially designed invertible networks that are easily trainable yet still powerful BIBREF16 , BIBREF17 , BIBREF19   Inspired by these works, we use the invertible transformation proposed by BIBREF16 , which consists of a series of “coupling layers”  This architecture is specially designed to guarantee a unit Jacobian determinant (and thus the invertibility property) 
From Eq  ( EQREF22 ) we know that only INLINEFORM0 is required for accomplishing learning and inference; we never need to explicitly construct INLINEFORM1   Thus, we directly define the architecture of INLINEFORM2   As shown in Figure FIGREF24 , the nonlinear transformation from the observed embedding INLINEFORM3 to INLINEFORM4 represents the first coupling layer  The input in this layer is partitioned into left and right halves of dimensions, INLINEFORM5 and INLINEFORM6 , respectively  A single coupling layer is defined as: DISPLAYFORM0
where INLINEFORM0 is the coupling function and can be any nonlinear form  This transformation satisfies INLINEFORM1 , and BIBREF16 show that its Jacobian matrix is triangular with all ones on the main diagonal  Thus the Jacobian determinant is always equal to one (i e  volume-preserving) and the invertibility condition is naturally satisfied 
To be sufficiently expressive, we compose multiple coupling layers as suggested in BIBREF16   Specifically, we exchange the role of left and right half vectors at each layer as shown in Figure FIGREF24   For instance, from INLINEFORM0 to INLINEFORM1 the left subset INLINEFORM2 is unchanged, while from INLINEFORM3 to INLINEFORM4 the right subset INLINEFORM5 remains the same  Also note that composing multiple coupling layers does not change the volume-preserving and invertibility properties  Such a sequence of invertible transformations from the data space INLINEFORM6 to INLINEFORM7 is also called normalizing flow BIBREF20  
Experiments
In this section, we first describe our datasets and experimental setup  We then instantiate our approach with Markov and DMV-structured syntax models, and report results on POS tagging and dependency grammar induction respectively  Lastly, we analyze the learned latent embeddings 
Data
For both POS tagging and dependency parsing, we run experiments on the Wall Street Journal (WSJ) portion of the Penn Treebank  To create the observed data embeddings, we train skip-gram word embeddings BIBREF7 that are found to capture syntactic properties well when trained with small context window BIBREF8 , BIBREF9   Following BIBREF9 , the dimensionality INLINEFORM0 is set to 100, and the training context window size is set to 1 to encode more syntactic information  The skip-gram embeddings are trained on the one billion word language modeling benchmark dataset BIBREF21 in addition to the WSJ corpus 
General Experimental Setup
For the neural projector, we employ rectified networks as coupling function INLINEFORM0 following BIBREF16   We use a rectified network with an input layer, one hidden layer, and linear output units, the number of hidden units is set to the same as the number of input units  The number of coupling layers are varied as 4, 8, 16 for both tasks  We optimize marginal data likelihood directly using Adam BIBREF22   For both tasks in the fully unsupervised setting, we do not tune the hyper-parameters using supervised data 
Unsupervised POS tagging
For unsupervised POS tagging, we use a Markov-structured syntax model in our approach, which is a popular structure for unsupervised tagging tasks BIBREF9 , BIBREF10  
Following existing literature, we train and test on the entire WSJ corpus (49208 sentences, 1M tokens)  We use 45 tag clusters, the number of POS tags that appear in WSJ corpus  We train the discrete HMM and the Gaussian HMM BIBREF9 as baselines  For the Gaussian HMM, mean vectors of Gaussian emissions are initialized with the empirical mean of all word vectors with an additive noise  We assume diagonal covariance matrix for INLINEFORM0 and initialize it with the empirical variance of the word vectors  Following BIBREF9 , the covariance matrix is fixed during training  The multinomial probabilities are initialized as INLINEFORM1 , where INLINEFORM2   For our approach, we initialize the syntax model and Gaussian parameters with the pre-trained Gaussian HMM  The weights of layers in the rectified network are initialized from a uniform distribution with mean zero and a standard deviation of INLINEFORM3 , where INLINEFORM4 is the input dimension  We evaluate the performance of POS tagging with both Many-to-One (M-1) accuracy BIBREF23 and V-Measure (VM) BIBREF24   Given a model we found that the tagging performance is well-correlated with the training data likelihood, thus we use training data likelihood as a unsupervised criterion to select the trained model over 10 random restarts after training 50 epochs  We repeat this process 5 times and report the mean and standard deviation of performance 
We compare our approach with basic HMM, Gaussian HMM, and several state-of-the-art systems, including sophisticated HMM variants and clustering techniques with hand-engineered features  The results are presented in Table TABREF32   Through the introduced latent embeddings and additional neural projection, our approach improves over the Gaussian HMM by 5 4 points in M-1 and 5 6 points in VM  Neural HMM (NHMM) BIBREF10 is a baseline that also learns word representation jointly  Both their basic model and extended Conv version does not outperform the Gaussian HMM  Their best model incorporates another LSTM to model long distance dependency and breaks the Markov assumption, yet our approach still achieves substantial improvement over it without considering more context information  Moreover, our method outperforms the best published result that benefits from hand-engineered features BIBREF27 by 2 0 points on VM 
We found that most tagging errors happen in noun subcategories  Therefore, we do the one-to-one mapping between gold POS tags and induced clusters and plot the normalized confusion matrix of noun subcategories in Figure FIGREF35   The Gaussian HMM fails to identify “NN” and “NNS” correctly for most cases, and it often recognizes “NNPS” as “NNP”  In contrast, our approach corrects these errors well 
Unsupervised Dependency Parsing without gold POS tags
For the task of unsupervised dependency parse induction, we employ the Dependency Model with Valence (DMV) BIBREF2 as the syntax model in our approach  DMV is a generative model that defines a probability distribution over dependency parse trees and syntactic categories, generating tokens and dependencies in a head-outward fashion  While, traditionally, DMV is trained using gold POS tags as observed syntactic categories, in our approach, we treat each tag as a latent variable, as described in sec:general-neural 
Most existing approaches to this task are not fully unsupervised since they rely on gold POS tags following the original experimental setup for DMV  This is partially because automatically parsing from words is difficult even when using unsupervised syntactic categories BIBREF29   However, inducing dependencies from words alone represents a more realistic experimental condition since gold POS tags are often unavailable in practice  Previous work that has trained from words alone often requires additional linguistic constraints (like sentence internal boundaries) BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , acoustic cues BIBREF33 , additional training data BIBREF4 , or annotated data from related languages BIBREF34   Our approach is naturally designed to train on word embeddings directly, thus we attempt to induce dependencies without using gold POS tags or other extra linguistic information 
Like previous work we use sections 02-21 of WSJ corpus as training data and evaluate on section 23, we remove punctuations and train the models on sentences of length INLINEFORM0 , “head-percolation” rules BIBREF39 are applied to obtain gold dependencies for evaluation  We train basic DMV, extended DMV (E-DMV) BIBREF35 and Gaussian DMV (which treats POS tag as unknown latent variables and generates observed word embeddings directly conditioned on them following Gaussian distribution) as baselines  Basic DMV and E-DMV are trained with Viterbi EM BIBREF40 on unsupervised POS tags induced from our Markov-structured model described in sec:pos  Multinomial parameters of the syntax model in both Gaussian DMV and our model are initialized with the pre-trained DMV baseline  Other parameters are initialized in the same way as in the POS tagging experiment  The directed dependency accuracy (DDA) is used for evaluation and we report accuracy on sentences of length INLINEFORM1 and all lengths  We train the parser until training data likelihood converges, and report the mean and standard deviation over 20 random restarts 
Our model directly observes word embeddings and does not require gold POS tags during training  Thus, results from related work trained on gold tags are not directly comparable  However, to measure how these systems might perform without gold tags, we run three recent state-of-the-art systems in our experimental setting: UR-A E-DMV BIBREF36 , Neural E-DMV BIBREF11 , and CRF Autoencoder (CRFAE) BIBREF37   We use unsupervised POS tags (induced from our Markov-structured model) in place of gold tags  We also train basic DMV on gold tags and include several state-of-the-art results on gold tags as reference points 
As shown in Table TABREF39 , our approach is able to improve over the Gaussian DMV by 4 8 points on length INLINEFORM0 and 4 8 points on all lengths, which suggests the additional latent embedding layer and neural projector are helpful  The proposed approach yields, to the best of our knowledge, state-of-the-art performance without gold POS annotation and without sentence-internal boundary information  DMV, UR-A E-DMV, Neural E-DMV, and CRFAE suffer a large decrease in performance when trained on unsupervised tags – an effect also seen in previous work BIBREF29 , BIBREF34   Since our approach induces latent POS tags jointly with dependency trees, it may be able to learn POS clusters that are more amenable to grammar induction than the unsupervised tags  We observe that CRFAE underperforms its gold-tag counterpart substantially  This may largely be a result of the model's reliance on prior linguistic rules that become unavailable when gold POS tag types are unknown  Many extensions to DMV can be considered orthogonal to our approach – they essentially focus on improving the syntax model  It is possible that incorporating these more sophisticated syntax models into our approach may lead to further improvements 
Sensitivity Analysis
In the above experiments we initialize the structured syntax components with the pre-trained Gaussian or discrete baseline, which is shown as a useful technique to help train our deep models  We further study the results with fully random initialization  In the POS tagging experiment, we report the results in Table TABREF48   While the performance with 4 layers is comparable to the pre-trained Gaussian initialization, deeper projections (8 or 16 layers) result in a dramatic drop in performance  This suggests that the structured syntax model with very deep projections is difficult to train from scratch, and a simpler projection might be a good compromise in the random initialization setting 
Different from the Markov prior in POS tagging experiments, our parsing model seems to be quite sensitive to the initialization  For example, directed accuracy of our approach on sentences of length INLINEFORM0 is below 40 0 with random initialization  This is consistent with previous work that has noted the importance of careful initialization for DMV-based models such as the commonly used harmonic initializer BIBREF2   However, it is not straightforward to apply the harmonic initializer for DMV directly in our model without using some kind of pre-training since we do not observe gold POS 
We investigate the effect of the choice of pre-trained embedding on performance while using our approach  To this end, we additionally include results using fastText embeddings BIBREF41 – which, in contrast with skip-gram embeddings, include character-level information  We set the context windows size to 1 and the dimension size to 100 as in the skip-gram training, while keeping other parameters set to their defaults  These results are summarized in Table TABREF50 and Table TABREF51   While fastText embeddings lead to reduced performance with our model, our approach still yields an improvement over the Gaussian baseline with the new observed embeddings space 
Qualitative Analysis of Embeddings
We perform qualitative analysis to understand how the latent embeddings help induce syntactic structures  First we filter out low-frequency words and punctuations in WSJ, and visualize the rest words (10k) with t-SNE BIBREF42 under different embeddings  We assign each word with its most likely gold POS tags in WSJ and color them according to the gold POS tags 
For our Markov-structured model, we have displayed the embedding space in Figure SECREF5 , where the gold POS clusters are well-formed  Further, we present five example target words and their five nearest neighbors in terms of cosine similarity  As shown in Table TABREF53 , the skip-gram embedding captures both semantic and syntactic aspects to some degree, yet our embeddings are able to focus especially on the syntactic aspects of words, in an unsupervised fashion without using any extra morphological information 
In Figure FIGREF54 we depict the learned latent embeddings with the DMV-structured syntax model  Unlike the Markov structure, the DMV structure maps a large subset of singular and plural nouns to the same overlapping region  However, two clusters of singular and plural nouns are actually separated  We inspect the two clusters and the overlapping region in Figure FIGREF54 , it turns out that the nouns in the separated clusters are words that can appear as subjects and, therefore, for which verb agreement is important to model  In contrast, the nouns in the overlapping region are typically objects  This demonstrates that the latent embeddings are focusing on aspects of language that are specifically important for modeling dependency without ever having seen examples of dependency parses  Some previous work has deliberately created embeddings to capture different notions of similarity BIBREF43 , BIBREF44 , while they use extra morphology or dependency annotations to guide the embedding learning, our approach provides a potential alternative to create new embeddings that are guided by structured syntax model, only using unlabeled text corpora 
Related Work
Our approach is related to flow-based generative models, which are first described in NICE BIBREF16 and have recently received more attention BIBREF17 , BIBREF19 , BIBREF18   This relevant work mostly adopts simple (e g  Gaussian) and fixed priors and does not attempt to learn interpretable latent structures  Another related generative model class is variational auto-encoders (VAEs) BIBREF45 that optimize a lower bound on the marginal data likelihood, and can be extended to learn latent structures BIBREF46 , BIBREF47   Against the flow-based models, VAEs remove the invertibility constraint but sacrifice the merits of exact inference and exact log likelihood computation, which potentially results in optimization challenges BIBREF48   Our approach can also be viewed in connection with generative adversarial networks (GANs) BIBREF49 that is a likelihood-free framework to learn implicit generative models  However, it is non-trivial for a gradient-based method like GANs to propagate gradients through discrete structures 
Conclusion
In this work, we define a novel generative approach to leverage continuous word representations for unsupervised learning of syntactic structure  Experiments on both POS induction and unsupervised dependency parsing tasks demonstrate the effectiveness of our proposed approach  Future work might explore more sophisticated invertible projections, or recurrent projections that jointly transform the entire input sequence 

ground_truth: Yes

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they evaluate only on English datasets?
Context 4: 
Source Data
As a starting point, we used the DIP corpus BIBREF37 , a collection of 49 clusters of 100 web pages on educational topics (e g  bullying, homeschooling, drugs) with a short description of each topic  It was created from a large web crawl using state-of-the-art information retrieval  We selected 30 of the topics for which we created the necessary concept map annotations 
Proposition Extraction
As concept maps consist of propositions expressing the relation between concepts (see Figure FIGREF2 ), we need to impose such a structure upon the plain text in the document clusters  This could be done by manually annotating spans representing concepts and relations, however, the size of our clusters makes this a huge effort: 2288 sentences per topic (69k in total) need to be processed  Therefore, we resort to an automatic approach 
The Open Information Extraction paradigm BIBREF38 offers a representation very similar to the desired one  For instance, from
Students with bad credit history should not lose hope and apply for federal loans with the FAFSA 
Open IE systems extract tuples of two arguments and a relation phrase representing propositions:
(s  with bad credit history, should not lose, hope)
(s  with bad credit history, apply for, federal loans with the FAFSA)
While the relation phrase is similar to a relation in a concept map, many arguments in these tuples represent useful concepts  We used Open IE 4, a state-of-the-art system BIBREF39 to process all sentences  After removing duplicates, we obtained 4137 tuples per topic 
Since we want to create a gold-standard corpus, we have to ensure that we produce high-quality data  We therefore made use of the confidence assigned to every extracted tuple to filter out low quality ones  To ensure that we do not filter too aggressively (and miss important aspects in the final summary), we manually annotated 500 tuples sampled from all topics for correctness  On the first 250 of them, we tuned the filter threshold to 0 5, which keeps 98 7% of the correct extractions in the unseen second half  After filtering, a topic had on average 2850 propositions (85k in total) 
Proposition Filtering
Despite the similarity of the Open IE paradigm, not every extracted tuple is a suitable proposition for a concept map  To reduce the effort in the subsequent steps, we therefore want to filter out unsuitable ones  A tuple is suitable if it (1) is a correct extraction, (2) is meaningful without any context and (3) has arguments that represent proper concepts  We created a guideline explaining when to label a tuple as suitable for a concept map and performed a small annotation study  Three annotators independently labeled 500 randomly sampled tuples  The agreement was 82% ( INLINEFORM0 )  We found tuples to be unsuitable mostly because they had unresolvable pronouns, conflicting with (2), or arguments that were full clauses or propositions, conflicting with (3), while (1) was mostly taken care of by the confidence filtering in § SECREF21  
Due to the high number of tuples we decided to automate the filtering step  We trained a linear SVM on the majority voted annotations  As features, we used the extraction confidence, length of arguments and relations as well as part-of-speech tags, among others  To ensure that the automatic classification does not remove suitable propositions, we tuned the classifier to avoid false negatives  In particular, we introduced class weights, improving precision on the negative class at the cost of a higher fraction of positive classifications  Additionally, we manually verified a certain number of the most uncertain negative classifications to further improve performance  When 20% of the classifications are manually verified and corrected, we found that our model trained on 350 labeled instances achieves 93% precision on negative classifications on the unseen 150 instances  We found this to be a reasonable trade-off of automation and data quality and applied the model to the full dataset 
The classifier filtered out 43% of the propositions, leaving 1622 per topic  We manually examined the 17k least confident negative classifications and corrected 955 of them  We also corrected positive classifications for certain types of tuples for which we knew the classifier to be imprecise  Finally, each topic was left with an average of 1554 propositions (47k in total) 
Importance Annotation
Given the propositions identified in the previous step, we now applied our crowdsourcing scheme as described in § SECREF4 to determine their importance  To cope with the large number of propositions, we combine the two task designs: First, we collect Likert-scores from 5 workers for each proposition, clean the data and calculate average scores  Then, using only the top 100 propositions according to these scores, we crowdsource 10% of all possible pairwise comparisons among them  Using TrueSkill, we obtain a fine-grained ranking of the 100 most important propositions 
For Likert-scores, the average agreement over all topics is 0 80, while the majority agreement for comparisons is 0 78  We repeated the data collection for three randomly selected topics and found the Pearson correlation between both runs to be 0 73 (Spearman 0 73) for Likert-scores and 0 72 (Spearman 0 71) for comparisons  These figures show that the crowdsourcing approach works on this dataset as reliably as on the TAC documents 
In total, we uploaded 53k scoring and 12k comparison tasks to Mechanical Turk, spending $4425 45 including fees  From the fine-grained ranking of the 100 most important propositions, we select the top 50 per topic to construct a summary concept map in the subsequent steps 
Proposition Revision
Having a manageable number of propositions, an annotator then applied a few straightforward transformations that correct common errors of the Open IE system  First, we break down propositions with conjunctions in either of the arguments into separate propositions per conjunct, which the Open IE system sometimes fails to do  And second, we correct span errors that might occur in the argument or relation phrases, especially when sentences were not properly segmented  As a result, we have a set of high quality propositions for our concept map, consisting of, due to the first transformation, 56 1 propositions per topic on average 
Concept Map Construction
In this final step, we connect the set of important propositions to form a graph  For instance, given the following two propositions
(student, may borrow, Stafford Loan)
(the student, does not have, a credit history)
one can easily see, although the first arguments differ slightly, that both labels describe the concept student, allowing us to build a concept map with the concepts student, Stafford Loan and credit history  The annotation task thus involves deciding which of the available propositions to include in the map, which of their concepts to merge and, when merging, which of the available labels to use  As these decisions highly depend upon each other and require context, we decided to use expert annotators rather than crowdsource the subtasks 
Annotators were given the topic description and the most important, ranked propositions  Using a simple annotation tool providing a visualization of the graph, they could connect the propositions step by step  They were instructed to reach a size of 25 concepts, the recommended maximum size for a concept map BIBREF6   Further, they should prefer more important propositions and ensure connectedness  When connecting two propositions, they were asked to keep the concept label that was appropriate for both propositions  To support the annotators, the tool used ADW BIBREF40 , a state-of-the-art approach for semantic similarity, to suggest possible connections  The annotation was carried out by graduate students with a background in NLP after receiving an introduction into the guidelines and tool and annotating a first example 
If an annotator was not able to connect 25 concepts, she was allowed to create up to three synthetic relations with freely defined labels, making the maps slightly abstractive  On average, the constructed maps have 0 77 synthetic relations, mostly connecting concepts whose relation is too obvious to be explicitly stated in text (e g  between Montessori teacher and Montessori education) 
To assess the reliability of this annotation step, we had the first three maps created by two annotators  We casted the task of selecting propositions to be included in the map as a binary decision task and observed an agreement of 84% ( INLINEFORM0 )  Second, we modeled the decision which concepts to join as a binary decision on all pairs of common concepts, observing an agreement of 95% ( INLINEFORM1 )  And finally, we compared which concept labels the annotators decided to include in the final map, observing 85% ( INLINEFORM2 ) agreement  Hence, the annotation shows substantial agreement BIBREF41  
Corpus Analysis
In this section, we describe our newly created corpus, which, in addition to having summaries in the form of concept maps, differs from traditional summarization corpora in several aspects 
Document Clusters
The corpus consists of document clusters for 30 different topics  Each of them contains around 40 documents with on average 2413 tokens, which leads to an average cluster size of 97,880 token  With these characteristics, the document clusters are 15 times larger than typical DUC clusters of ten documents and five times larger than the 25-document-clusters (Table TABREF26 )  In addition, the documents are also more variable in terms of length, as the (length-adjusted) standard deviation is twice as high as in the other corpora  With these properties, the corpus represents an interesting challenge towards real-world application scenarios, in which users typically have to deal with much more than ten documents 
Because we used a large web crawl as the source for our corpus, it contains documents from a variety of genres  To further analyze this property, we categorized a sample of 50 documents from the corpus  Among them, we found professionally written articles and blog posts (28%), educational material for parents and kids (26%), personal blog posts (16%), forum discussions and comments (12%), commented link collections (12%) and scientific articles (6%) 
In addition to the variety of genres, the documents also differ in terms of language use  To capture this property, we follow Zopf 2016 and compute, for every topic, the average Jensen-Shannon divergence between the word distribution of one document and the word distribution in the remaining documents  The higher this value is, the more the language differs between documents  We found the average divergence over all topics to be 0 3490, whereas it is 0 3019 in DUC 2004 and 0 3188 in TAC 2008A 
Concept Maps
As Table TABREF33 shows, each of the 30 reference concept maps has exactly 25 concepts and between 24 and 28 relations  Labels for both concepts and relations consist on average of 3 2 tokens, whereas the latter are a bit shorter in characters 
To obtain a better picture of what kind of text spans have been used as labels, we automatically tagged them with their part-of-speech and determined their head with a dependency parser  Concept labels tend to be headed by nouns (82%) or verbs (15%), while they also contain adjectives, prepositions and determiners  Relation labels, on the other hand, are almost always headed by a verb (94%) and contain prepositions, nouns and particles in addition  These distributions are very similar to those reported by Villalon 2010 for their (single-document) concept map corpus 
Analyzing the graph structure of the maps, we found that all of them are connected  They have on average 7 2 central concepts with more than one relation, while the remaining ones occur in only one proposition  We found that achieving a higher number of connections would mean compromising importance, i e  including less important propositions, and decided against it 
Baseline Experiments
In this section, we briefly describe a baseline and evaluation scripts that we release, with a detailed documentation, along with the corpus 
Conclusion
In this work, we presented low-context importance annotation, a novel crowdsourcing scheme that we used to create a new benchmark corpus for concept-map-based MDS  The corpus has large-scale document clusters of heterogeneous web documents, posing a challenging summarization task  Together with the corpus, we provide implementations of a baseline method and evaluation scripts and hope that our efforts facilitate future research on this variant of summarization 
Acknowledgments
We would like to thank Teresa Botschen, Andreas Hanselowski and Markus Zopf for their help with the annotation work and Christian Meyer for his valuable feedback  This work has been supported by the German Research Foundation as part of the Research Training Group “Adaptive Preparation of Information from Heterogeneous Sources” (AIPHES) under grant No  GRK 1994/1 

ground_truth: Yes

Unanswerable