prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they report results only on English data?
Context 1: Introduction
Distributed word representations, commonly referred to as word embeddings BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , serve as elementary building blocks in the course of algorithm design for an expanding range of applications in natural language processing (NLP), including named entity recognition BIBREF4 , BIBREF5 , parsing BIBREF6 , sentiment analysis BIBREF7 , BIBREF8 , and word-sense disambiguation BIBREF9   Although the empirical utility of word embeddings as an unsupervised method for capturing the semantic or syntactic features of a certain word as it is used in a given lexical resource is well-established BIBREF10 , BIBREF11 , BIBREF12 , an understanding of what these features mean remains an open problem BIBREF13 , BIBREF14 and as such word embeddings mostly remain a black box  It is desirable to be able to develop insight into this black box and be able to interpret what it means, while retaining the utility of word embeddings as semantically-rich intermediate representations  Other than the intrinsic value of this insight, this would not only allow us to explain and understand how algorithms work BIBREF15 , but also set a ground that would facilitate the design of new algorithms in a more deliberate way 
Recent approaches to generating word embeddings (e g  BIBREF0 , BIBREF2 ) are rooted linguistically in the field of distributed semantics BIBREF16 , where words are taken to assume meaning mainly by their degree of interaction (or lack thereof) with other words in the lexicon BIBREF17 , BIBREF18   Under this paradigm, dense, continuous vector representations are learned in an unsupervised manner from a large corpus, using the word cooccurrence statistics directly or indirectly, and such an approach is shown to result in vector representations that mathematically capture various semantic and syntactic relations between words BIBREF0 , BIBREF2 , BIBREF3   However, the dense nature of the learned embeddings obfuscate the distinct concepts encoded in the different dimensions, which renders the resulting vectors virtually uninterpretable  The learned embeddings make sense only in relation to each other and their specific dimensions do not carry explicit information that can be interpreted  However, being able to interpret a word embedding would illuminate the semantic concepts implicitly represented along the various dimensions of the embedding, and reveal its hidden semantic structures 
In the literature, researchers tackled interpretability problem of the word embeddings using different approaches  Several researchers BIBREF19 , BIBREF20 , BIBREF21 proposed algorithms based on non-negative matrix factorization (NMF) applied to cooccurrence variant matrices  Other researchers suggested to obtain interpretable word vectors from existing uninterpretable word vectors by applying sparse coding BIBREF22 , BIBREF23 , by training a sparse auto-encoder to transform the embedding space BIBREF24 , by rotating the original embeddings BIBREF25 , BIBREF26 or by applying transformations based on external semantic datasets BIBREF27  
Although the above-mentioned approaches provide better interpretability that is measured using a particular method such as word intrusion test, usually the improved interpretability comes with a cost of performance in the benchmark tests such as word similarity or word analogy  One possible explanation for this performance decrease is that the proposed transformations from the original embedding space distort the underlying semantic structure constructed by the original embedding algorithm  Therefore, it can be claimed that a method that learns dense and interpretable word embeddings without inflicting any damage to the underlying semantic learning mechanism is the key to achieve both high performing and interpretable word embeddings 
Especially after the introduction of the word2vec algorithm by Mikolov BIBREF0 , BIBREF1 , there has been a growing interest in algorithms that generate improved word representations under some performance metric  Significant effort is spent on appropriately modifying the objective functions of the algorithms in order to incorporate knowledge from external resources, with the purpose of increasing the performance of the resulting word representations BIBREF28 , BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 , BIBREF37   Inspired by the line of work reported in these studies, we propose to use modified objective functions for a different purpose: learning more interpretable dense word embeddings  By doing this, we aim to incorporate semantic information from an external lexical resource into the word embedding so that the embedding dimensions are aligned along predefined concepts  This alignment is achieved by introducing a modification to the embedding learning process  In our proposed method, which is built on top of the GloVe algorithm BIBREF2 , the cost function for any one of the words of concept word-groups is modified by the introduction of an additive term to the cost function  Each embedding vector dimension is first associated with a concept  For a word belonging to any one of the word-groups representing these concepts, the modified cost term favors an increase for the value of this word's embedding vector dimension corresponding to the concept that the particular word belongs to  For words that do not belong to any one of the word-groups, the cost term is left untouched  Specifically, Roget's Thesaurus BIBREF38 , BIBREF39 is used to derive the concepts and concept word-groups to be used as the external lexical resource for our proposed method  We quantitatively demonstrate the increase in interpretability by using the measure given in BIBREF27 , BIBREF40 as well as demonstrating qualitative results  We also show that the semantic structure of the original embedding has not been harmed in the process since there is no performance loss with standard word-similarity or word-analogy tests 
The paper is organized as follows  In Section SECREF2 , we discuss previous studies related to our work under two main categories: interpretability of word embeddings and joint-learning frameworks where the objective function is modified  In Section SECREF3 , we present the problem framework and provide the formulation within the GloVe BIBREF2 algorithm setting  In Section SECREF4 where our approach is proposed, we motivate and develop a modification to the original objective function with the aim of increasing representation interpretability  In Section SECREF5 , experimental results are provided and the proposed method is quantitatively and qualitatively evaluated  Additionally, in Section SECREF5 , results demonstrating the extent to which the original semantic structure of the embedding space is affected are presented by using word-analogy and word-similarity tests  We conclude the paper in Section SECREF6  
Related Work
Methodologically, our work is related to prior studies that aim to obtain “improved” word embeddings using external lexical resources, under some performance metric  Previous work in this area can be divided into two main categories: works that i) modify the word embedding learning algorithm to incorporate lexical information, ii) operate on pre-trained embeddings with a post-processing step 
Among works that follow the first approach, BIBREF28 extend the Skip-Gram model by incorporating the word similarity relations extracted from the Paraphrase Database (PPDB) and WordNet BIBREF29 , into the Skip-Gram predictive model as an additional cost term  In BIBREF30 , the authors extend the CBOW model by considering two types of semantic information, termed relational and categorical, to be incorporated into the embeddings during training  For the former type of semantic information, the authors propose the learning of explicit vectors for the different relations extracted from a semantic lexicon such that the word pairs that satisfy the same relation are distributed more homogeneously  For the latter, the authors modify the learning objective such that some weighted average distance is minimized for words under the same semantic category  In BIBREF31 , the authors represent the synonymy and hypernymy-hyponymy relations in terms of inequality constraints, where the pairwise similarity rankings over word triplets are forced to follow an order extracted from a lexical resource  Following their extraction from WordNet, the authors impose these constraints in the form of an additive cost term to the Skip-Gram formulation  Finally, BIBREF32 builds on top of the GloVe algorithm by introducing a regularization term to the objective function that encourages the vector representations of similar words as dictated by WordNet to be similar as well 
Turning our attention to the post-processing approach for enriching word embeddings with external lexical knowledge, BIBREF33 has introduced the retrofitting algorithm that acts on pre-trained embeddings such as Skip-Gram or GloVe  The authors propose an objective function that aims to balance out the semantic information captured in the pre-trained embeddings with the constraints derived from lexical resources such as WordNet, PPDB and FrameNet  One of the models proposed in BIBREF34 extends the retrofitting approach to incorporate the word sense information from WordNet  Similarly, BIBREF35 creates multi-sense embeddings by gathering the word sense information from a lexical resource and learning to decompose the pre-trained embeddings into a convex combination of sense embeddings  In BIBREF36 , the authors focus on improving word embeddings for capturing word similarity, as opposed to mere relatedness  To this end, they introduce the counter-fitting technique which acts on the input word vectors such that synonymous words are attracted to one another whereas antonymous words are repelled, where the synonymy-antonymy relations are extracted from a lexical resource  More recently, the ATTRACT-REPEL algorithm proposed by BIBREF37 improves on counter-fitting by a formulation which imparts the word vectors with external lexical information in mini-batches 
Most of the studies discussed above ( BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF36 , BIBREF37 ) report performance improvements in benchmark tests such as word similarity or word analogy, while BIBREF29 uses a different analysis method (mean reciprocal rank)  In sum, the literature is rich with studies aiming to obtain word embeddings that perform better under specific performance metrics  However, less attention has been directed to the issue of interpretability of the word embeddings  In the literature, the problem of interpretability has been tackled using different approaches  BIBREF19 proposed non-negative matrix factorization (NMF) for learning sparse, interpretable word vectors from co-occurrence variant matrices where the resulting vector space is called non-negative sparse embeddigns (NNSE)  However, since NMF methods require maintaining a global matrix for learning, they suffer from memory and scale issue  This problem has been addressed in BIBREF20 where an online method of learning interpretable word embeddings from corpora using a modified version of skip-gram model BIBREF0 is proposed  As a different approach, BIBREF21 combined text-based similarity information among words with brain activity based similarity information to improve interpretability using joint non-negative sparse embedding (JNNSE) 
A common alternative approach for learning interpretable embeddings is to learn transformations that map pre-trained state-of-the-art embeddings to new interpretable semantic spaces  To obtain sparse, higher dimensional and more interpretable vector spaces, BIBREF22 and BIBREF23 use sparse coding on conventional dense word embeddings  However, these methods learn the projection vectors that are used for the transformation from the word embeddings without supervision  For this reason, labels describing the corresponding semantic categories cannot be provided  An alternative approach was proposed in BIBREF25 , where orthogonal transformations were utilized to increase interpretability while preserving the performance of the underlying embedding  However, BIBREF25 has also shown that total interpretability of an embedding is kept constant under any orthogonal transformation and it can only be redistributed across the dimensions  Rotation algorithms based on exploratory factor analysis (EFA) to preserve the performance of the original word embeddings while improving their interpretability was proposed in BIBREF26   BIBREF24 proposed to deploy a sparse auto-encoder using pre-trained dense word embeddings to improve interpretability  More detailed investigation of semantic structure and interpretability of word embeddings can be found in BIBREF27 , where a metric was proposed to quantitatively measure the degree of interpretability already present in the embedding vector spaces 
Previous works on interpretability mentioned above, except BIBREF21 , BIBREF27 and our proposed method, do not need external resources, utilization of which has both advantages and disadvantages  Methods that do not use external resources require fewer resources but they also lack the aid of information extracted from these resources 
Problem Description
For the task of unsupervised word embedding extraction, we operate on a discrete collection of lexical units (words) INLINEFORM0 that is part of an input corpus INLINEFORM1 , with number of tokens INLINEFORM2 , sourced from a vocabulary INLINEFORM3 of size INLINEFORM4   In the setting of distributional semantics, the objective of a word embedding algorithm is to maximize some aggregate utility over the entire corpus so that some measure of “closeness” is maximized for pairs of vector representations INLINEFORM14 for words which, on the average, appear in proximity to one another  In the GloVe algorithm BIBREF2 , which we base our improvements upon, the following objective function is considered: DISPLAYFORM0
In ( EQREF6 ), INLINEFORM0 and INLINEFORM1 stand for word and context vector representations, respectively, for words INLINEFORM2 and INLINEFORM3 , while INLINEFORM4 represents the (possibly weighted) cooccurrence count for the word pair INLINEFORM5   Intuitively, ( EQREF6 ) represents the requirement that if some word INLINEFORM6 occurs often enough in the context (or vicinity) of another word INLINEFORM7 , then the corresponding word representations should have a large enough inner product in keeping with their large INLINEFORM8 value, up to some bias terms INLINEFORM9 ; and vice versa  INLINEFORM10 in ( EQREF6 ) is used as a discounting factor that prohibits rare cooccurrences from disproportionately influencing the resulting embeddings 
The objective ( EQREF6 ) is minimized using stochastic gradient descent by iterating over the matrix of cooccurrence records INLINEFORM0   In the GloVe algorithm, for a given word INLINEFORM1 , the final word representation is taken to be the average of the two intermediate vector representations obtained from ( EQREF6 ); i e, INLINEFORM2   In the next section, we detail the enhancements made to ( EQREF6 ) for the purposes of enhanced interpretability, using the aforementioned framework as our basis 
Imparting Interpretability
Our approach falls into a joint-learning framework where the distributional information extracted from the corpus is allowed to fuse with the external lexicon-based information  Word-groups extracted from Roget's Thesaurus are directly mapped to individual dimensions of word embeddings  Specifically, the vector representations of words that belong to a particular group are encouraged to have deliberately increased values in a particular dimension that corresponds to the word-group under consideration  This can be achieved by modifying the objective function of the embedding algorithm to partially influence vector representation distributions across their dimensions over an input vocabulary  To do this, we propose the following modification to the GloVe objective in ( EQREF6 ): rCl J = i,j=1V f(Xij)[ (wiTwj + bi + bj -Xij)2
+ k(l=1D INLINEFORM0 iFl g(wi,l) + l=1D INLINEFORM1 j Fl g(wj,l) ) ]  In ( SECREF4 ), INLINEFORM2 denotes the indices for the elements of the INLINEFORM3 th concept word-group which we wish to assign in the vector dimension INLINEFORM4   The objective ( SECREF4 ) is designed as a mixture of two individual cost terms: the original GloVe cost term along with a second term that encourages embedding vectors of a given concept word-group to achieve deliberately increased values along an associated dimension INLINEFORM5   The relative weight of the second term is controlled by the parameter INLINEFORM6   The simultaneous minimization of both objectives ensures that words that are similar to, but not included in, one of these concept word-groups are also “nudged” towards the associated dimension INLINEFORM7   The trained word vectors are thus encouraged to form a distribution where the individual vector dimensions align with certain semantic concepts represented by a collection of concept word-groups, one assigned to each vector dimension  To facilitate this behaviour, ( SECREF4 ) introduces a monotone decreasing function INLINEFORM8 defined as INLINEFORM9
which serves to increase the total cost incurred if the value of the INLINEFORM0 th dimension for the two vector representations INLINEFORM1 and INLINEFORM2 for a concept word INLINEFORM3 with INLINEFORM4 fails to be large enough  INLINEFORM5 is also shown in Fig  FIGREF7  
The objective ( SECREF4 ) is minimized using stochastic gradient descent over the cooccurrence records INLINEFORM0   Intuitively, the terms added to ( SECREF4 ) in comparison with ( EQREF6 ) introduce the effect of selectively applying a positive step-type input to the original descent updates of ( EQREF6 ) for concept words along their respective vector dimensions, which influences the dimension value in the positive direction  The parameter INLINEFORM1 in ( SECREF4 ) allows for the adjustment of the magnitude of this influence as needed 
In the next section, we demonstrate the feasibility of this approach by experiments with an example collection of concept word-groups extracted from Roget's Thesaurus 
Experiments and Results
We first identified 300 concepts, one for each dimension of the 300-dimensional vector representation, by employing Roget's Thesaurus  This thesaurus follows a tree structure which starts with a Root node that contains all the words and phrases in the thesaurus  The root node is successively split into Classes and Sections, which are then (optionally) split into Subsections of various depths, finally ending in Categories, which constitute the smallest unit of word/phrase collections in the structure  The actual words and phrases descend from these Categories, and make up the leaves of the tree structure  We note that a given word typically appears in multiple categories corresponding to the different senses of the word  We constructed concept word-groups from Roget's Thesaurus as follows: We first filtered out the multi-word phrases and the relatively obscure terms from the thesaurus  The obscure terms were identified by checking them against a vocabulary extracted from Wikipedia  We then obtained 300 word-groups as the result of a partitioning operation applied to the subtree that ends with categories as its leaves  The partition boundaries, hence the resulting word-groups, can be chosen in many different ways  In our proposed approach, we have chosen to determine this partitioning by traversing this tree structure from the root node in breadth-first order, and by employing a parameter INLINEFORM0 for the maximum size of a node  Here, the size of a node is defined as the number of unique words that ever-descend from that node  During the traversal, if the size of a given node is less than this threshold, we designate the words that ultimately descend from that node as a concept word-group  Otherwise, if the node has children, we discard the node, and queue up all its children for further consideration  If this node does not have any children, on the other hand, the node is truncated to INLINEFORM1 elements with the highest frequency-ranks, and the resulting words are designated as a concept word-group  We note that the choice of INLINEFORM2 greatly affects the resulting collection of word-groups: Excessively large values result in few word-groups that greatly overlap with one another, while overly small values result in numerous tiny word-groups that fail to adequately represent a concept  We experimentally determined that a INLINEFORM3 value of 452 results in the most healthy number of relatively large word-groups (113 groups with size INLINEFORM4 100), while yielding a preferably small overlap amongst the resulting word-groups (with average overlap size not exceeding 3 words)  A total of 566 word-groups were thus obtained  259 smallest word-groups (with size INLINEFORM5 38) were discarded to bring down the number of word-groups to 307  Out of these, 7 groups with the lowest median frequency-rank were further discarded, which yields the final 300 concept word-groups used in the experiments  We present some of the resulting word-groups in Table TABREF9  
By using the concept word-groups, we have trained the GloVe algorithm with the proposed modification given in Section SECREF4 on a snapshot of English Wikipedia measuring 8GB in size, with the stop-words filtered out  Using the parameters given in Table TABREF10 , this resulted in a vocabulary size of 287,847  For the weighting parameter in Eq  SECREF4 , we used a value of INLINEFORM0   The algorithm was trained over 20 iterations  The GloVe algorithm without any modifications was also trained as a baseline with the same parameters  In addition to the original GloVe algorithm, we compare our proposed method with previous studies that aim to obtain interpretable word vectors  We train the improved projected gradient model proposed in BIBREF20 to obtain word vectors (called OIWE-IPG) using the same corpus we use to train GloVe and our proposed method  Using the methods proposed in BIBREF23 , BIBREF26 , BIBREF24 on our baseline GloVe embeddings, we obtain SOV, SPINE and Parsimax (orthogonal) word representations, respectively  We train all the models with the proposed parameters  However, in BIBREF26 , the authors show results for a relatively small vocabulary of 15,000 words  When we trained their model on our baseline GloVe embeddings with a large vocabulary of size 287,847, the resulting vectors performed significantly poor on word similarity tasks compared to the results presented in their paper  In addition, Parsimax (orthogonal) word vectors obtained using method in BIBREF26 are nearly identical to the baseline vectors (i e  learned orthogonal transformation matrix is very close to identity)  Therefore, Parsimax (orthogonal) yields almost same results with baseline vectors in all evaluations  We evaluate the interpretability of the resulting embeddings qualitatively and quantitatively  We also test the performance of the embeddings on word similarity and word analogy tests 
In our experiments, vocabulary size is close to 300,000 while only 16,242 unique words of the vocabulary are present in the concept groups  Furthermore, only dimensions that correspond to the concept group of the word will be updated due to the additional cost term  Given that these concept words can belong to multiple concept groups (2 on average), only 33,319 parameters are updated  There are 90 million individual parameters present for the 300,000 word vectors of size 300  Of these parameters, only approximately 33,000 are updated by the additional cost term 
Qualitative Evaluation for Interpretability
In Fig  FIGREF13 , we demonstrate the particular way in which the proposed algorithm ( SECREF4 ) influences the vector representation distributions  Specifically, we consider, for illustration, the 32nd dimension values for the original GloVe algorithm and our modified version, restricting the plots to the top-1000 words with respect to their frequency ranks for clarity of presentation  In Fig  FIGREF13 , the words in the horizontal axis are sorted in descending order with respect to the values at the 32nd dimension of their word embedding vectors coming from the original GloVe algorithm  The dimension values are denoted with blue and red/green markers for the original and the proposed algorithms, respectively  Additionally, the top-50 words that achieve the greatest 32nd dimension values among the considered 1000 words are emphasized with enlarged markers, along with text annotations  In the presented simulation of the proposed algorithm, the 32nd dimension values are encoded with the concept JUDGMENT, which is reflected as an increase in the dimension values for words such as committee, academy, and article  We note that these words (red) are not part of the pre-determined word-group for the concept JUDGMENT, in contrast to words such as award, review and account (green) which are  This implies that the increase in the corresponding dimension values seen for these words is attributable to the joint effect of the first term in ( SECREF4 ) which is inherited from the original GloVe algorithm, in conjunction with the remaining terms in the proposed objective expression ( SECREF4 )  This experiment illustrates that the proposed algorithm is able to impart the concept of JUDGMENT on its designated vector dimension above and beyond the supplied list of words belonging to the concept word-group for that dimension  We also present the list of words with the greatest dimension value for the dimensions 11, 13, 16, 31, 36, 39, 41, 43 and 79 in Table TABREF11   These dimensions are aligned/imparted with the concepts that are given in the column headers  In Table TABREF11 , the words that are highlighted with green denote the words that exist in the corresponding word-group obtained from Roget's Thesaurus (and are thus explicitly forced to achieve increased dimension values), while the red words denote the words that achieve increased dimension values by virtue of their cooccurrence statistics with the thesaurus-based words (indirectly, without being explicitly forced)  This again illustrates that a semantic concept can indeed be coded to a vector dimension provided that a sensible lexical resource is used to guide semantically related words to the desired vector dimension via the proposed objective function in ( SECREF4 )  Even the words that do not appear in, but are semantically related to, the word-groups that we formed using Roget's Thesaurus, are indirectly affected by the proposed algorithm  They also reflect the associated concepts at their respective dimensions even though the objective functions for their particular vectors are not modified  This point cannot be overemphasized  Although the word-groups extracted from Roget's Thesaurus impose a degree of supervision to the process, the fact that the remaining words in the entire vocabulary are also indirectly affected makes the proposed method a semi-supervised approach that can handle words that are not in these chosen word-groups  A qualitative example of this result can be seen in the last column of Table TABREF11   It is interesting to note the appearance of words such as guerilla, insurgency, mujahideen, Wehrmacht and Luftwaffe in addition to the more obvious and straightforward army, soldiers and troops, all of which are not present in the associated word-group WARFARE 
Most of the dimensions we investigated exhibit similar behaviour to the ones presented in Table TABREF11   Thus generally speaking, we can say that the entries in Table TABREF11 are representative of the great majority  However, we have also specifically looked for dimensions that make less sense and determined a few such dimensions which are relatively less satisfactory  These less satisfactory examples are given in Table TABREF14   These examples are also interesting in that they shed insight into the limitations posed by polysemy and existence of very rare outlier words 
Quantitative Evaluation for Interpretability
One of the main goals of this study is to improve the interpretability of dense word embeddings by aligning the dimensions with predefined concepts from a suitable lexicon  A quantitative measure is required to reliably evaluate the achieved improvement  One of the methods proposed to measure the interpretability is the word intrusion test BIBREF41   But, this method is expensive to apply since it requires evaluations from multiple human evaluators for each embedding dimension  In this study, we use a semantic category-based approach based on the method and category dataset (SEMCAT) introduced in BIBREF27 to quantify interpretability  Specifically, we apply a modified version of the approach presented in BIBREF40 in order to consider possible sub-groupings within the categories in SEMCAT  Interpretability scores are calculated using Interpretability Score (IS) as given below:
DISPLAYFORM0
In ( EQREF17 ), INLINEFORM0 and INLINEFORM1 represents the interpretability scores in the positive and negative directions of the INLINEFORM2 dimension ( INLINEFORM3 , INLINEFORM4 number of dimensions in the embedding space) of word embedding space for the INLINEFORM5 category ( INLINEFORM6 , INLINEFORM7 is number of categories in SEMCAT, INLINEFORM8 ) in SEMCAT respectively  INLINEFORM9 is the set of words in the INLINEFORM10 category in SEMCAT and INLINEFORM11 is the number of words in INLINEFORM12   INLINEFORM13 corresponds to the minimum number of words required to construct a semantic category (i e  represent a concept)  INLINEFORM14 represents the set of INLINEFORM15 words that have the highest ( INLINEFORM16 ) and lowest ( INLINEFORM17 ) values in INLINEFORM18 dimension of the embedding space  INLINEFORM19 is the intersection operator and INLINEFORM20 is the cardinality operator (number of elements) for the intersecting set  In ( EQREF17 ), INLINEFORM21 gives the interpretability score for the INLINEFORM22 dimension and INLINEFORM23 gives the average interpretability score of the embedding space 
Fig  FIGREF18 presents the measured average interpretability scores across dimensions for original GloVe embeddings, for the proposed method and for the other four methods we compare, along with a randomly generated embedding  Results are calculated for the parameters INLINEFORM0 and INLINEFORM1   Our proposed method significantly improves the interpretability for all INLINEFORM2 compared to the original GloVe approach  Our proposed method is second to only SPINE in increasing interpretability  However, as we will experimentally demonstrate in the next subsection, in doing this, SPINE almost entirely destroys the underlying semantic structure of the word embeddings, which is the primary function of a word embedding 
The proposed method and interpretability measurements are both based on utilizing concepts represented by word-groups  Therefore it is expected that there will be higher interpretability scores for some of the dimensions for which the imparted concepts are also contained in SEMCAT  However, by design, word groups that they use are formed by using different sources and are independent  Interpretability measurements use SEMCAT while our proposed method utilizes Roget's Thesaurus 
Intrinsic Evaluation of the Embeddings
It is necessary to show that the semantic structure of the original embedding has not been damaged or distorted as a result of aligning the dimensions with given concepts, and that there is no substantial sacrifice involved from the performance that can be obtained with the original GloVe  To check this, we evaluate performances of the proposed embeddings on word similarity BIBREF42 and word analogy BIBREF0 tests  We compare the results with the original embeddings and the three alternatives excluding Parsimax BIBREF26 since orthogonal transformations will not affect the performance of the original embeddings on these tests 
Word similarity test measures the correlation between word similarity scores obtained from human evaluation (i e  true similarities) and from word embeddings (usually using cosine similarity)  In other words, this test quantifies how well the embedding space reflects human judgements in terms of similarities between different words  The correlation scores for 13 different similarity test sets are reported in Table TABREF20   We observe that, let alone a reduction in performance, the obtained scores indicate an almost uniform improvement in the correlation values for the proposed algorithm, outperforming all the alternatives in almost all test sets  Categories from Roget's thesaurus are groupings of words that are similar in some sense which the original embedding algorithm may fail to capture  These test results signify that the semantic information injected into the algorithm by the additional cost term is significant enough to result in a measurable improvement  It should also be noted that scores obtained by SPINE is unacceptably low on almost all tests indicating that it has achieved its interpretability performance at the cost of losing its semantic functions 
Word analogy test is introduced in BIBREF1 and looks for the answers of the questions that are in the form of "X is to Y, what Z is to ?" by applying simple arithmetic operations to vectors of words X, Y and Z  We present precision scores for the word analogy tests in Table TABREF21   It can be seen that the alternative approaches that aim to improve interpretability, have poor performance on the word analogy tests  However, our proposed method has comparable performance with the original GloVe embeddings  Our method outperforms GloVe in semantic analogy test set and in overall results, while GloVe performs slightly better in syntactic test set  This comparable performance is mainly due to the cost function of our proposed method that includes the original objective of the GloVe 
To investigate the effect of the additional cost term on the performance improvement in the semantic analogy test, we present Table TABREF22   In particular, we present results for the cases where i) all questions in the dataset are considered, ii) only the questions that contains at least one concept word are considered, iii) only the questions that consist entirely of concept words are considered  We note specifically that for the last case, only a subset of the questions under the semantic category family txt ended up being included  We observe that for all three scenarios, our proposed algorithm results in an improvement in the precision scores  However, the greatest performance increase is seen for the last scenario, which underscores the extent to which the semantic features captured by embeddings can be improved with a reasonable selection of the lexical resource from which the concept word-groups were derived 
Conclusion
We presented a novel approach to impart interpretability into word embeddings  We achieved this by encouraging different dimensions of the vector representation to align with predefined concepts, through the addition of an additional cost term in the optimization objective of the GloVe algorithm that favors a selective increase for a pre-specified input of concept words along each dimension 
We demonstrated the efficacy of this approach by applying qualitative and quantitative evaluations for interpretability  We also showed via standard word-analogy and word-similarity tests that the semantic coherence of the original vector space is preserved, even slightly improved  We have also performed and reported quantitative comparisons with several other methods for both interpretabilty increase and preservation of semantic coherence  Upon inspection of Fig  FIGREF18 and Tables TABREF20 , TABREF21 , and TABREF22 altogether, it should be noted that our proposed method achieves both of the objectives simultaneously, increased interpretability and preservation of the intrinsic semantic structure 
An important point was that, while it is expected for words that are already included in the concept word-groups to be aligned together since their dimensions are directly updated with the proposed cost term, it was also observed that words not in these groups also aligned in a meaningful manner without any direct modification to their cost function  This indicates that the cost term we added works productively with the original cost function of GloVe to handle words that are not included in the original concept word-groups, but are semantically related to those word-groups  The underlying mechanism can be explained as follows  While the outside lexical resource we introduce contains a relatively small number of words compared to the total number of words, these words and the categories they represent have been carefully chosen and in a sense, "densely span" all the words in the language  By saying "span", we mean they cover most of the concepts and ideas in the language without leaving too many uncovered areas  With "densely" we mean all areas are covered with sufficient strength  In other words, this subset of words is able to constitute a sufficiently strong skeleton, or scaffold  Now remember that GloVe works to align or bring closer related groups of words, which will include words from the lexical source  So the joint action of aligning the words with the predefined categories (introduced by us) and aligning related words (handled by GloVe) allows words not in the lexical groups to also be aligned meaningfully  We may say that the non-included words are "pulled along" with the included words by virtue of the "strings" or "glue" that is provided by GloVe  In numbers, the desired effect is achieved by manipulating less than only 0 05% of parameters of the entire word vectors  Thus, while there is a degree of supervision coming from the external lexical resource, the rest of the vocabulary is also aligned indirectly in an unsupervised way  This may be the reason why, unlike earlier proposed approaches, our method is able to achieve increasing interpretability without destroying underlying semantic structure, and consequently without sacrificing performance in benchmark tests 
Upon inspecting the 2nd column of Table TABREF14 , where qualitative results for concept TASTE are presented, another insight regarding the learning mechanism of our proposed approach can be made  Here it seems understandable that our proposed approach, along with GloVe, brought together the words taste and polish, and then the words Polish and, for instance, Warsaw are brought together by GloVe  These examples are interesting in that they shed insight into how GloVe works and the limitations posed by polysemy  It should be underlined that the present approach is not totally incapable of handling polysemy, but cannot do so perfectly  Since related words are being clustered, sufficiently well-connected words that do not meaningfully belong along with others will be appropriately "pulled away" from that group by several words, against the less effective, inappropriate pull of a particular word  Even though polish with lowercase "p" belongs where it is, it is attracting Warsaw to itself through polysemy and this is not meaningful  Perhaps because Warsaw is not a sufficiently well-connected word, it ends being dragged along, although words with greater connectedness to a concept group might have better resisted such inappropriate attractions 
In this study, we used the GloVe algorithm as the underlying dense word embedding scheme to demonstrate our approach  However, we stress that it is possible for our approach to be extended to other word embedding algorithms which have a learning routine consisting of iterations over cooccurrence records, by making suitable adjustments in the objective function  Since word2vec model is also based on the coocurrences of words in a sliding window through a large corpus, we expect that our approach can also be applied to word2vec after making suitable adjustments, which can be considered as an immediate future work for our approach  Although the semantic concepts are encoded in only one direction (positive) within the embedding dimensions, it might be beneficial to pursue future work that also encodes opposite concepts, such as good and bad, in two opposite directions of the same dimension 
The proposed methodology can also be helpful in computational cross-lingual studies, where the similarities are explored across the vector spaces of different languages BIBREF43 , BIBREF44  
Context 2: Introduction
Word Sense Disambiguation (WSD) is a fundamental task and long-standing challenge in Natural Language Processing (NLP), which aims to find the exact sense of an ambiguous word in a particular context BIBREF0  Previous WSD approaches can be grouped into two main categories: knowledge-based and supervised methods 
Knowledge-based WSD methods rely on lexical resources like WordNet BIBREF1 and usually exploit two kinds of lexical knowledge  The gloss, which defines a word sense meaning, is first utilized in Lesk algorithm BIBREF2 and then widely taken into account in many other approaches BIBREF3, BIBREF4  Besides, structural properties of semantic graphs are mainly used in graph-based algorithms BIBREF5, BIBREF6 
Traditional supervised WSD methods BIBREF7, BIBREF8, BIBREF9 focus on extracting manually designed features and then train a dedicated classifier (word expert) for every target lemma 
Although word expert supervised WSD methods perform better, they are less flexible than knowledge-based methods in the all-words WSD task BIBREF10  Recent neural-based methods are devoted to dealing with this problem  BIBREF11 present a supervised classifier based on Bi-LSTM, which shares parameters among all word types except the last layer  BIBREF10 convert WSD task to a sequence labeling task, thus building a unified model for all polysemous words  However, neither of them can totally beat the best word expert supervised methods 
More recently, BIBREF12 propose to leverage the gloss information from WordNet and model the semantic relationship between the context and gloss in an improved memory network  Similarly, BIBREF13 introduce a (hierarchical) co-attention mechanism to generate co-dependent representations for the context and gloss  Their attempts prove that incorporating gloss knowledge into supervised WSD approach is helpful, but they still have not achieved much improvement, because they may not make full use of gloss knowledge 
In this paper, we focus on how to better leverage gloss information in a supervised neural WSD system  Recently, the pre-trained language models, such as ELMo BIBREF14 and BERT BIBREF15, have shown their effectiveness to alleviate the effort of feature engineering  Especially, BERT has achieved excellent results in question answering (QA) and natural language inference (NLI)  We construct context-gloss pairs from glosses of all possible senses (in WordNet) of the target word, thus treating WSD task as a sentence-pair classification problem  We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task  In particular, our contribution is two-fold:
1  We construct context-gloss pairs and propose three BERT-based models for WSD 
2  We fine-tune the pre-trained BERT model, and the experimental results on several English all-words WSD benchmark datasets show that our approach significantly outperforms the state-of-the-art systems 
Methodology
In this section, we describe our method in detail 
Methodology ::: Task Definition
In WSD, a sentence $s$ usually consists of a series of words: $\lbrace w_1,\cdots ,w_m\rbrace $, and some of the words $\lbrace w_{i_1},\cdots ,w_{i_k}\rbrace $ are targets $\lbrace t_1,\cdots ,t_k\rbrace $ need to be disambiguated  For each target $t$, its candidate senses $\lbrace c_1,\cdots ,c_n\rbrace $ come from entries of its lemma in a pre-defined sense inventory (usually WordNet)  Therefore, WSD task aims to find the most suitable entry (symbolized as unique sense key) for each target in a sentence  See a sentence example in Table TABREF1 
Methodology ::: BERT
BERT BIBREF15 is a new language representation model, and its architecture is a multi-layer bidirectional Transformer encoder  BERT model is pre-trained on a large corpus and two novel unsupervised prediction tasks, i e , masked language model and next sentence prediction tasks are used in pre-training  When incorporating BERT into downstream tasks, the fine-tuning procedure is recommended  We fine-tune the pre-trained BERT model on WSD task 
Methodology ::: BERT ::: BERT(Token-CLS)
Since every target in a sentence needs to be disambiguated to find its exact sense, WSD task can be regarded as a token-level classification task  To incorporate BERT to WSD task, we take the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer for every target lemma, which is the same as the last layer of the Bi-LSTM model BIBREF11 
Methodology ::: GlossBERT
BERT can explicitly model the relationship of a pair of texts, which has shown to be beneficial to many pair-wise natural language understanding tasks  In order to fully leverage gloss information, we propose GlossBERT to construct context-gloss pairs from all possible senses of the target word in WordNet, thus treating WSD task as a sentence-pair classification problem 
We describe our construction method with an example (See Table TABREF1)  There are four targets in this sentence, and here we take target word research as an example:
Methodology ::: GlossBERT ::: Context-Gloss Pairs
The sentence containing target words is denoted as context sentence  For each target word, we extract glosses of all $N$ possible senses (here $N=4$) of the target word (research) in WordNet to obtain the gloss sentence  [CLS] and [SEP] marks are added to the context-gloss pairs to make it suitable for the input of BERT model  A similar idea is also used in aspect-based sentiment analysis BIBREF16 
Methodology ::: GlossBERT ::: Context-Gloss Pairs with Weak Supervision
Based on the previous construction method, we add weak supervised signals to the context-gloss pairs (see the highlighted part in Table TABREF1)  The signal in the gloss sentence aims to point out the target word, and the signal in the context sentence aims to emphasize the target word considering the situation that a target word may occur more than one time in the same sentence 
Therefore, each target word has $N$ context-gloss pair training instances ($label\in \lbrace yes, no\rbrace $)  When testing, we output the probability of $label=yes$ of each context-gloss pair and choose the sense corresponding to the highest probability as the prediction label of the target word  We experiment with three GlossBERT models:
Methodology ::: GlossBERT ::: GlossBERT(Token-CLS)
We use context-gloss pairs as input  We highlight the target word by taking the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer ($label\in \lbrace yes, no\rbrace $) 
Methodology ::: GlossBERT ::: GlossBERT(Sent-CLS)
We use context-gloss pairs as input  We take the final hidden state of the first token [CLS] as the representation of the whole sequence and add a classification layer ($label\in \lbrace yes, no\rbrace $), which does not highlight the target word 
Methodology ::: GlossBERT ::: GlossBERT(Sent-CLS-WS)
We use context-gloss pairs with weak supervision as input  We take the final hidden state of the first token [CLS] and add a classification layer ($label\in \lbrace yes, no\rbrace $), which weekly highlight the target word by the weak supervision 
Experiments ::: Datasets
The statistics of the WSD datasets are shown in Table TABREF12 
Experiments ::: Datasets ::: Training Dataset
Following previous work BIBREF13, BIBREF12, BIBREF10, BIBREF17, BIBREF9, BIBREF7, we choose SemCor3 0 as training corpus, which is the largest corpus manually annotated with WordNet sense for WSD 
Experiments ::: Datasets ::: Evaluation Datasets
We evaluate our method on several English all-words WSD datasets  For a fair comparison, we use the benchmark datasets proposed by BIBREF17 which include five standard all-words fine-grained WSD datasets from the Senseval and SemEval competitions: Senseval-2 (SE2), Senseval-3 (SE3), SemEval-2007 (SE07), SemEval-2013 (SE13) and SemEval-2015 (SE15)  Following BIBREF13, BIBREF12 and BIBREF10, we choose SE07, the smallest among these test sets, as the development set 
Experiments ::: Datasets ::: WordNet
Since BIBREF17 map all the sense annotations in these datasets from their original versions to WordNet 3 0, we extract word sense glosses from WordNet 3 0 
Experiments ::: Settings
We use the pre-trained uncased BERT$_\mathrm {BASE}$ model for fine-tuning, because we find that BERT$_\mathrm {LARGE}$ model performs slightly worse than BERT$_\mathrm {BASE}$ in this task  The number of Transformer blocks is 12, the number of the hidden layer is 768, the number of self-attention heads is 12, and the total number of parameters of the pre-trained model is 110M  When fine-tuning, we use the development set (SE07) to find the optimal settings for our experiments  We keep the dropout probability at 0 1, set the number of epochs to 4  The initial learning rate is 2e-5, and the batch size is 64 
Experiments ::: Results
Table TABREF19 shows the performance of our method on the English all-words WSD benchmark datasets  We compare our approach with previous methods 
The first block shows the MFS baseline, which selects the most frequent sense in the training corpus for each target word 
The second block shows two knowledge-based systems  Lesk$_{ext+emb}$ BIBREF4 is a variant of Lesk algorithm BIBREF2 by calculating the gloss-context overlap of the target word  Babelfy BIBREF6 is a unified graph-based approach which exploits the semantic network structure from BabelNet 
The third block shows two word expert traditional supervised systems  IMS BIBREF7 is a flexible framework which trains SVM classifiers and uses local features  And IMS$_{+emb}$ BIBREF9 is the best configuration of the IMS framework, which also integrates word embeddings as features 
The fourth block shows several recent neural-based methods  Bi-LSTM BIBREF11 is a baseline for neural models  Bi-LSTM$_{+ att  + LEX + POS}$ BIBREF10 is a multi-task learning framework for WSD, POS tagging, and LEX with self-attention mechanism, which converts WSD to a sequence learning task  GAS$_{ext}$ BIBREF12 is a variant of GAS which is a gloss-augmented variant of the memory network by extending gloss knowledge  CAN$^s$ and HCAN BIBREF13 are sentence-level and hierarchical co-attention neural network models which leverage gloss knowledge 
In the last block, we report the performance of our method  BERT(Token-CLS) is our baseline, which does not incorporate gloss information, and it performs slightly worse than previous traditional supervised methods and recent neural-based methods  It proves that directly using BERT cannot obtain performance growth  The other three methods outperform other models by a substantial margin, which proves that the improvements come from leveraging BERT to better exploit gloss information  It is worth noting that our method achieves significant improvements in SE07 and Verb over previous methods, which have the highest ambiguity level among all datasets and all POS tags respectively according to BIBREF17 
Moreover, GlossBERT(Token-CLS) performs better than GlossBERT(Sent-CLS), which proves that highlighting the target word in the sentence is important  However, the weakly highlighting method GlossBERT(Sent-CLS-WS) performs best in most circumstances, which may result from its combination of the advantages of the other two methods 
Experiments ::: Discussion
There are two main reasons for the great improvements of our experimental results  First, we construct context-gloss pairs and convert WSD problem to a sentence-pair classification task which is similar to NLI tasks and train only one classifier, which is equivalent to expanding the corpus  Second, we leverage BERT BIBREF15 to better exploit the gloss information  BERT model shows its advantage in dealing with sentence-pair classification tasks by its amazing improvement on QA and NLI tasks  This advantage comes from both of its two novel unsupervised prediction tasks 
Compared with traditional word expert supervised methods, our GlossBERT shows its effectiveness to alleviate the effort of feature engineering and does not require training a dedicated classifier for every target lemma  Up to now, it can be said that the neural network method can totally beat the traditional word expert method  Compared with recent neural-based methods, our solution is more intuitive and can make better use of gloss knowledge  Besides, our approach demonstrates that when we fine-tune BERT on a downstream task, converting it into a sentence-pair classification task may be a good choice 
Conclusion
In this paper, we seek to better leverage gloss knowledge in a supervised neural WSD system  We propose a new solution to WSD by constructing context-gloss pairs and then converting WSD to a sentence-pair classification task  We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task 
Acknowledgments
We would like to thank the anonymous reviewers for their valuable comments  The research work is supported by National Natural Science Foundation of China (No  61751201 and 61672162), Shanghai Municipal Science and Technology Commission (16JC1420401 and 17JC1404100), Shanghai Municipal Science and Technology Major Project (No 2018SHZDZX01) and ZJLab 

ground_truth: Yes

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they use large or small BERT?
Context 3: Introduction
Grammar induction is the task of inducing hierarchical syntactic structure from data  Statistical approaches to grammar induction require specifying a probabilistic grammar (e g  formalism, number and shape of rules), and fitting its parameters through optimization  Early work found that it was difficult to induce probabilistic context-free grammars (PCFG) from natural language data through direct methods, such as optimizing the log likelihood with the EM algorithm BIBREF0 , BIBREF1   While the reasons for the failure are manifold and not completely understood, two major potential causes are the ill-behaved optimization landscape and the overly strict independence assumptions of PCFGs  More successful approaches to grammar induction have thus resorted to carefully-crafted auxiliary objectives BIBREF2 , priors or non-parametric models BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , and manually-engineered features BIBREF7 , BIBREF8 to encourage the desired structures to emerge 
We revisit these aforementioned issues in light of advances in model parameterization and inference  First, contrary to common wisdom, we find that parameterizing a PCFG's rule probabilities with neural networks over distributed representations makes it possible to induce linguistically meaningful grammars by simply optimizing log likelihood  While the optimization problem remains non-convex, recent work suggests that there are optimization benefits afforded by over-parameterized models BIBREF9 , BIBREF10 , BIBREF11 , and we indeed find that this neural PCFG is significantly easier to optimize than the traditional PCFG  Second, this factored parameterization makes it straightforward to incorporate side information into rule probabilities through a sentence-level continuous latent vector, which effectively allows different contexts in a derivation to coordinate  In this compound PCFG—continuous mixture of PCFGs—the context-free assumptions hold conditioned on the latent vector but not unconditionally, thereby obtaining longer-range dependencies within a tree-based generative process 
To utilize this approach, we need to efficiently optimize the log marginal likelihood of observed sentences  While compound PCFGs break efficient inference, if the latent vector is known the distribution over trees reduces to a standard PCFG  This property allows us to perform grammar induction using a collapsed approach where the latent trees are marginalized out exactly with dynamic programming  To handle the latent vector, we employ standard amortized inference using reparameterized samples from a variational posterior approximated from an inference network BIBREF12 , BIBREF13  
On standard benchmarks for English and Chinese, the proposed approach is found to perform favorably against recent neural network-based approaches to grammar induction BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17  
Probabilistic Context-Free Grammars
We consider context-free grammars (CFG) consisting of a 5-tuple INLINEFORM0 where INLINEFORM1 is the distinguished start symbol, INLINEFORM2 is a finite set of nonterminals, INLINEFORM3 is a finite set of preterminals, INLINEFORM6 is a finite set of terminal symbols, and INLINEFORM7 is a finite set of rules of the form,
INLINEFORM0
A probabilistic context-free grammar (PCFG) consists of a grammar INLINEFORM0 and rule probabilities INLINEFORM1 such that INLINEFORM2 is the probability of the rule INLINEFORM3   Letting INLINEFORM4 be the set of all parse trees of INLINEFORM5 , a PCFG defines a probability distribution over INLINEFORM6 via INLINEFORM7 where INLINEFORM8 is the set of rules used in the derivation of INLINEFORM9   It also defines a distribution over string of terminals INLINEFORM10 via
INLINEFORM0
where INLINEFORM0 , i e  the set of trees INLINEFORM1 such that INLINEFORM2 's leaves are INLINEFORM3   We will slightly abuse notation and use
INLINEFORM0
to denote the posterior distribution over the unobserved latent trees given the observed sentence INLINEFORM0 , where INLINEFORM1 is the indicator function 
Compound PCFGs
A compound probability distribution BIBREF19 is a distribution whose parameters are themselves random variables  These distributions generalize mixture models to the continuous case, for example in factor analysis which assumes the following generative process,
INLINEFORM0
Compound distributions provide the ability to model rich generative processes, but marginalizing over the latent parameter can be computationally intractable unless conjugacy can be exploited 
In this work, we study compound probabilistic context-free grammars whose distribution over trees arises from the following generative process: we first obtain rule probabilities via
INLINEFORM0
where INLINEFORM0 is a prior with parameters INLINEFORM1 (spherical Gaussian in this paper), and INLINEFORM2 is a neural network that concatenates the input symbol embeddings with INLINEFORM3 and outputs the sentence-level rule probabilities INLINEFORM4 ,
INLINEFORM0
where INLINEFORM0 denotes vector concatenation  Then a tree/sentence is sampled from a PCFG with rule probabilities given by INLINEFORM1 ,
INLINEFORM0
This can be viewed as a continuous mixture of PCFGs, or alternatively, a Bayesian PCFG with a prior on sentence-level rule probabilities parameterized by INLINEFORM0   Importantly, under this generative model the context-free assumptions hold conditioned on INLINEFORM3 , but they do not hold unconditionally  This is shown in Figure FIGREF3 (right) where there is a dependence path through INLINEFORM4 if it is not conditioned upon  Compound PCFGs give rise to a marginal distribution over parse trees INLINEFORM5 via
INLINEFORM0
where INLINEFORM0   The subscript in INLINEFORM1 denotes the fact that the rule probabilities depend on INLINEFORM2   Compound PCFGs are clearly more expressive than PCFGs as each sentence has its own set of rule probabilities  However, it still assumes a tree-based generative process, making it possible to learn latent tree structures 
Our motivation for the compound PCFG is based on the observation that for grammar induction, context-free assumptions are generally made not because they represent an adequate model of natural language, but because they allow for tractable training  We can in principle model richer dependencies through vertical/horizontal Markovization BIBREF21 , BIBREF22 and lexicalization BIBREF23   However such dependencies complicate training due to the rapid increase in the number of rules  Under this view, we can interpret the compound PCFG as a restricted version of some lexicalized, higher-order PCFG where a child can depend on structural and lexical context through a shared latent vector  We hypothesize that this dependence among siblings is especially useful in grammar induction from words, where (for example) if we know that watched is used as a verb then the noun phrase is likely to be a movie 
In contrast to the usual Bayesian treatment of PCFGs which places priors on global rule probabilities BIBREF3 , BIBREF4 , BIBREF6 , the compound PCFG assumes a prior on local, sentence-level rule probabilities  It is therefore closely related to the Bayesian grammars studied by BIBREF25 and BIBREF26 , who also sample local rule probabilities from a logistic normal prior for training dependency models with valence (DMV) BIBREF27  
Experimental Setup
Results and Discussion
Table TABREF23 shows the unlabeled INLINEFORM0 scores for our models and various baselines  All models soundly outperform right branching baselines, and we find that the neural PCFG/compound PCFG are strong models for grammar induction  In particular the compound PCFG outperforms other models by an appreciable margin on both English and Chinese  We again note that we were unable to induce meaningful grammars through a traditional PCFG with the scalar parameterization despite a thorough hyperparameter search  See lab:full for the full results (including corpus-level INLINEFORM1 ) broken down by sentence length 
Table TABREF27 analyzes the learned tree structures  We compare similarity as measured by INLINEFORM0 against gold, left, right, and “self" trees (top), where self INLINEFORM1 score is calculated by averaging over all 6 pairs obtained from 4 different runs  We find that PRPN is particularly consistent across multiple runs  We also observe that different models are better at identifying different constituent labels, as measured by label recall (Table TABREF27 , bottom)  While left as future work, this naturally suggests an ensemble approach wherein the empirical probabilities of constituents (obtained by averaging the predicted binary constituent labels from the different models) are used either to supervise another model or directly as potentials in a CRF constituency parser  Finally, all models seemed to have some difficulty in identifying SBAR/VP constituents which typically span more words than NP constituents 
Related Work
Grammar induction has a long and rich history in natural language processing  Early work on grammar induction with pure unsupervised learning was mostly negative BIBREF0 , BIBREF1 , BIBREF74 , though BIBREF75 reported some success on partially bracketed data  BIBREF76 and BIBREF2 were some of the first successful statistical approaches to grammar induction  In particular, the constituent-context model (CCM) of BIBREF2 , which explicitly models both constituents and distituents, was the basis for much subsequent work BIBREF27 , BIBREF7 , BIBREF8   Other works have explored imposing inductive biases through Bayesian priors BIBREF4 , BIBREF5 , BIBREF6 , modified objectives BIBREF42 , and additional constraints on recursion depth BIBREF77 , BIBREF48  
While the framework of specifying the structure of a grammar and learning the parameters is common, other methods exist  BIBREF43 consider a nonparametric-style approach to unsupervised parsing by using random subsets of training subtrees to parse new sentences  BIBREF46 utilize an incremental algorithm to unsupervised parsing which makes local decisions to create constituents based on a complex set of heuristics  BIBREF47 induce parse trees through cascaded applications of finite state models 
More recently, neural network-based approaches to grammar induction have shown promising results on inducing parse trees directly from words  BIBREF14 , BIBREF15 learn tree structures through soft gating layers within neural language models, while BIBREF16 combine recursive autoencoders with the inside-outside algorithm  BIBREF17 train unsupervised recurrent neural network grammars with a structured inference network to induce latent trees, and BIBREF78 utilize image captions to identify and ground constituents 
Our work is also related to latent variable PCFGs BIBREF79 , BIBREF80 , BIBREF81 , which extend PCFGs to the latent variable setting by splitting nonterminal symbols into latent subsymbols  In particular, latent vector grammars BIBREF82 and compositional vector grammars BIBREF83 also employ continuous vectors within their grammars  However these approaches have been employed for learning supervised parsers on annotated treebanks, in contrast to the unsupervised setting of the current work 
Conclusion
This work explores grammar induction with compound PCFGs, which modulate rule probabilities with per-sentence continuous latent vectors  The latent vector induces marginal dependencies beyond the traditional first-order context-free assumptions within a tree-based generative process, leading to improved performance  The collapsed amortized variational inference approach is general and can be used for generative models which admit tractable inference through partial conditioning  Learning deep generative models which exhibit such conditional Markov properties is an interesting direction for future work 
Acknowledgments
We thank Phil Blunsom for initial discussions which seeded many of the core ideas in the present work  We also thank Yonatan Belinkov and Shay Cohen for helpful feedback, and Andrew Drozdov for providing the parsed dataset from their DIORA model  YK is supported by a Google Fellowship  AMR acknowledges the support of NSF 1704834, 1845664, AWS, and Oracle 
Model Parameterization
We associate an input embedding INLINEFORM0 for each symbol INLINEFORM1 on the left side of a rule (i e  INLINEFORM2 ) and run a neural network over INLINEFORM3 to obtain the rule probabilities  Concretely, each rule type INLINEFORM4 is parameterized as follows, INLINEFORM5
where INLINEFORM0 is the product space INLINEFORM1 , and INLINEFORM2 are MLPs with two residual layers, INLINEFORM3
The bias terms for the above expressions (including for the rule probabilities) are omitted for notational brevity  In Figure FIGREF3 we use the following to refer to rule probabilities of different rule types, INLINEFORM0
where INLINEFORM0 denotes the set of rules with INLINEFORM1 on the left hand side 
The compound PCFG rule probabilities INLINEFORM0 given a latent vector INLINEFORM1 , INLINEFORM2
Again the bias terms are omitted for brevity, and INLINEFORM0 are as before where the first layer's input dimensions are appropriately changed to account for concatenation with INLINEFORM1  
Corpus/Sentence F 1 F_1 by Sentence Length
For completeness we show the corpus-level and sentence-level INLINEFORM0 broken down by sentence length in Table TABREF44 , averaged across 4 different runs of each model 
Experiments with RNNGs
For experiments on supervising RNNGs with induced trees, we use the parameterization and hyperparameters from BIBREF17 , which uses a 2-layer 650-dimensional stack LSTM (with dropout of 0 5) and a 650-dimensional tree LSTM BIBREF88 , BIBREF90 as the composition function 
Concretely, the generative story is as follows: first, the stack representation is used to predict the next action (shift or reduce) via an affine transformation followed by a sigmoid  If shift is chosen, we obtain a distribution over the vocabulary via another affine transformation over the stack representation followed by a softmax  Then we sample the next word from this distribution and shift the generated word onto the stack using the stack LSTM  If reduce is chosen, we pop the last two elements off the stack and use the tree LSTM to obtain a new representation  This new representation is shifted onto the stack via the stack LSTM  Note that this RNNG parameterization is slightly different than the original from BIBREF53 , which does not ignore constituent labels and utilizes a bidirectional LSTM as the composition function instead of a tree LSTM  As our RNNG parameterization only works with binary trees, we binarize the gold trees with right binarization for the RNNG trained on gold trees (trees from the unsupervised methods explored in this paper are already binary)  The RNNG also trains a discriminative parser alongside the generative model for evaluation with importance sampling  We use a CRF parser whose span score parameterization is similar similar to recent works BIBREF89 , BIBREF87 , BIBREF85 : position embeddings are added to word embeddings, and a bidirectional LSTM with 256 hidden dimensions is run over the input representations to obtain the forward and backward hidden states  The score INLINEFORM0 for a constituent spanning the INLINEFORM1 -th and INLINEFORM2 -th word is given by,
INLINEFORM0
where the MLP has a single hidden layer with INLINEFORM0 nonlinearity followed by layer normalization BIBREF84  
For experiments on fine-tuning the RNNG with the unsupervised RNNG, we take the discriminative parser (which is also pretrained alongside the RNNG on induced trees) to be the structured inference network for optimizing the evidence lower bound  We refer the reader to BIBREF17 and their open source implementation for additional details  We also observe that as noted by BIBREF17 , a URNNG trained from scratch on this version of PTB without punctuation failed to outperform a right-branching baseline 
The LSTM language model baseline is the same size as the stack LSTM (i e  2 layers, 650 hidden units, dropout of 0 5), and is therefore equivalent to an RNNG with completely right branching trees  The PRPN/ON baselines for perplexity/syntactic evaluation in Table TABREF30 also have 2 layers with 650 hidden units and 0 5 dropout  Therefore all models considered in Table TABREF30 have roughly the same capacity  For all models we share input/output word embeddings BIBREF86   Perplexity estimation for the RNNGs and the compound PCFG uses 1000 importance-weighted samples 
For grammaticality judgment, we modify the publicly available dataset from BIBREF56 to only keep sentence pairs that did not have any unknown words with respect to our PTB vocabulary of 10K words  This results in 33K sentence pairs for evaluation 
Nonterminal/Preterminal Alignments
Figure FIGREF50 shows the part-of-speech alignments and Table TABREF46 shows the nonterminal label alignments for the compound PCFG/neural PCFG 
Subtree Analysis
Table TABREF53 lists more examples of constituents within each subtree as the top principical component is varied  Due to data sparsity, the subtree analysis is performed on the full dataset  See section UID36 for more details 

ground_truth: small BERT

small BERT

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. which chinese datasets were used?
Context 4:  The different models of word vectors perform similarly and thus one cannot privilege a certain type of word vectors  Interestingly, the clusters learned on the Wikipedia GloVe vectors offer competitive performance with respect to the in-domain word vectors used for the other cases showing that one can rely to out-of-domain data for constructing such representations 
Concerning the classification task (Table TABREF7 ) we generally observe a drop in the performance of the tagger as we deal with 10 classes  This essentially corresponds to a multi-class problem with 21 classes: one for the non-entity type and two classes for each entity type  In this setting we notice that the best results are obtained in most cases for higher number of classes (1000 or 2000) possibly due to a better discriminatory power in higher dimensions  Note also, that in some cases the addition of word cluster features does not necessarily improve the performance  Contrary, it may degrade it as it is evident in the case of glove INLINEFORM0 word clusters  Like in the case of segmentation we do not observe a word vector model that clearly outperforms the rest  Finally, we note the same competitive performance of the Wikipedia word clusters and notably for the glove INLINEFORM1 clusters which obtain the best F1-score 
Fine-grained Sentiment Analysis
The task of fine grained sentiment classification consists in predicting the sentiment of an input text according to a five point scale (sentiment INLINEFORM0 {VeryNegative, Negative, Neutral, Positive, VeryPositive})  We use the setting of task 4 of SemEval2016 “Sentiment Analysis in Twitter” and the dataset released by the organizers for subtask 4 BIBREF13  
In total, the training (resp  test) data consist of 9,070 (resp  20,632) tweets 
The evaluation measure selected in BIBREF13 for the task in the macro-averaged Mean Absolute Error (MAE INLINEFORM0 )  It is a measure of error, hence lower values are better  The measure's goal is to take into account the order of the classes when penalizing the decision of a classifier  For instance, misclassifying a very negative example as very positive is a bigger mistake than classifying it as negative or neutral  Penalizing a classifier according to how far the predictions are from the true class is captured by MAE INLINEFORM1 BIBREF14   Also, the advantage of using the macro- version instead of the standard version of the measure is the robustness against the class imbalance in the data 
Learning algorithm To demonstrate the efficiency of cluster membership features we rely on the system of BIBREF15 which was ranked 1st among 11 participants and uses a Logistic Regression as a learning algorithm  We follow the same feature extraction steps which consist of extracting n-gram and character n-gram features, part-of-speech counts as well as sentiment scores using standard sentiment lexicons such as the Bing Liu's BIBREF16 and the MPQA lexicons BIBREF17   For the full description, we refer the interested reader to BIBREF15  
To evaluate the performance of the proposed feature augmentation technique, we present in Table TABREF10 the macro-averaged Mean Absolute Error scores for different settings on the official test set of BIBREF13   First, notice that the best score in the test data is achieved using cluster membership features, where the word embeddings are trained using the skipgram model  The achieved score improves the state-of-the art on the dataset, which to the best of our knowledge was by BIBREF15   Also, note that the score on the test data improves for each type of embeddings used, which means that augmenting the feature space using cluster membership features helps the sentiment classification task 
Note, also, that using the clusters produced by the out-of-domain embeddings trained on wikipedia that were released as part of BIBREF8 performs surprisingly well  One might have expected their addition to hurt the performance  However, their value probably stems from the sheer amount of data used for their training as well as the relatively simple type of words (like awesome, terrible) which are discriminative for this task  Lastly, note that in each of the settings, the best results are achieved when the number of clusters is within INLINEFORM0 as in the NER tasks  Comparing the performance across the different embeddings, one cannot claim that a particular embedding performs better  It is evident though that augmenting the feature space with feature derived using the proposed method, preferably with in-domain data, helps the classification performance and reduces MAE INLINEFORM1  
From the results of Table TABREF10 it is clear that the addition of the cluster membership features improves the sentiment classification performance  To better understand though why these clusters help, we manually examined a sample of the words associated with the clusters  To improve the eligibility of those results we first removed the hashtags and we filter the results using an English vocabulary  In Table TABREF11 we present sample words from two of the most characteristic clusters with respect to the task of sentiment classification  Notice how words with positive and negative meanings are put in the respective clusters 
Fine-Grained Sentiment Quantification
Quantification is the problem of estimating the prevalence of a class in a dataset  While classification concerns assigning a category to a single instance, like labeling a tweet with the sentiment it conveys, the goal of quantification is, given a set of instances, to estimate the relative frequency of single class  Therefore, sentiment quantification tries to answer questions like “Given a set of tweets about the new iPhone, what is the fraction of VeryPositive ones?”  In the rest, we show the effect of the features derived from the word embeddings clusters in the fine-grained classification problem, which was also part of the SemEval-2016 “Sentiment Analysis in Twitter” task BIBREF13  
Learning Algorithm To perform the quantification task, we rely on a classify and count approach, which was shown effective in a related binary quantification problem BIBREF15   The idea is that given a set of instances on a particular subject, one first classifies the instances and then aggregates the counts  To this end, we use the same feature representation steps and data with the ones used for fine grained classification (Section 3 2)  Note that the data of the task are associated with subjects (described in full detail at BIBREF13 ), and, hence, quantification is performed for the tweets of a subject  For each of the five categories, the output of the approach is a 5-dimensional vector with the estimated prevalence of the categories 
The evaluation measure for the problem is the Earth Movers Distance (EMD) BIBREF18   EMD is a measure of error, hence lower values are better  It assumes ordered categories, which in our problem is naturally defined  Further assuming that the distance of consecutive categories (e g , Positive and VeryPositive) is 1, the measure is calculated by: INLINEFORM0
where INLINEFORM0 is number of categories (five in our case) and INLINEFORM1 and INLINEFORM2 are the true and predicted prevalence respectively BIBREF19  
Results Table TABREF13 presents the results of augmenting the feature set with the proposed features  We use Logistic Regression as a base classifier for the classify and count approach  Notice the positive impact of the features in the performance in the task  Adding the features derived from clustering the embeddings consistently improves the performance  Interestingly, the best performance ( INLINEFORM0 ) is achieved using the out-of-domain vectors, as in the NER classification task  Also, notice how the approach improves over the state-of-the-art performance in the challenge ( INLINEFORM1 ) BIBREF13 , held by the method of BIBREF20   The improvement over the method of BIBREF20 however, does not necessarily mean that classify and count performs better in the task  It implies that the feature set we used is richer, that in turn highlights the value of robust feature extraction mechanisms which is the subject of this paper 
Conclusion
We have shown empirically the effectiveness of incorporating cluster membership features in the feature extraction pipeline of Named-Entity recognition, sentiment classification and quantification tasks  Our results strongly suggest that incorporating cluster membership features benefit the performance in the tasks  The fact that the performance improvements are consistent in the four tasks we investigated, further highlights their usefulness, both for practitioners and researchers 
Although our study does not identify a clear winner with respect to the type of word vectors (skipgram, cbow, or GloVe), our findings suggest that one should first try skip-gram embeddings of low dimensionality ( INLINEFORM0 ) and high number of clusters (e g , INLINEFORM1 ) as the results obtained using these settings are consistently competitive  Our results also suggest that using out-of-domain data, like Wikipedia articles in this case, to construct the word embeddings is a good practice, as the results we obtained with these vectors are also competitive  The positive of out-of-domain embeddings and their combination with in-domain ones remains to be further studied 

ground_truth: Answer with content missing: (Data section) Chinese with version 5.1 of the Chinese Penn Treebank (CTB)