Paraphrase-Supervised Models of Compositionality

Compositional vector space models of meaning promise new solutions to stubborn language understanding problems. This paper makes two contributions toward this end: (i) it uses automatically-extracted paraphrase examples as a source of supervision for training compositional models, replacing previous work which relied on manual annotations used for the same purpose, and (ii) develops a context-aware model for scoring phrasal compositionality. Experimental results indicate that these multiple sources of information can be used to learn partial semantic supervision that matches previous techniques in intrinsic evaluation tasks. Our approaches are also evaluated for their impact on a machine translation system where we show improvements in translation quality, demonstrating that compositionality in interpretation correlates with compositionality in translation.

section_name: Introduction
paragraphs: Numerous lexical semantic properties are captured by representations encoding distributional properties of words, as has been demonstrated in a variety of tasks BIBREF0 , BIBREF1 , BIBREF2 . However, this distributional account of meaning does not scale to larger units like phrases and sentences BIBREF3 , BIBREF4 , motivating research into compositional models that combine word representations to produce representations of the semantics of longer units BIBREF5 , BIBREF6 , BIBREF7 . Previous work has learned these models using autoencoder formulations BIBREF8 or limited human supervision BIBREF5 . In this work, we explore the hypothesis that the equivalent knowledge about how words compose can be obtained through monolingual paraphrases that have been extracted using word alignments and an intermediate language BIBREF9 . Confirming this hypothesis would allow the rapid development of compositional models in a large number of languages.
As their name suggests, these models also impose the assumption that longer units like phrases are compositional, i.e., a phrase's meaning can be understood from the literal meaning of its parts. However, countless examples that run contrary to the assumption exist, and handling these non-compositional phrases has been problematic and of long-standing interest in the community BIBREF10 , BIBREF11 . (Non-) Compositionality detection can provide vital information to other language processing systems on whether a multiword unit should be treated semantically as a single entity or not, and scoring this phenomenon is particularly relevant for downstream tasks like machine translation (MT) or information retrieval. We explore the hypothesis that contextual evidence can be used to determine the relative degree to which a phrase is meant compositionally.
Rather than focusing purely on intrinsic clean-room evaluations, the goal of this work is to learn relatively accurate context-sensitive compositional models that are also directly applicable in real-world, noisy-data scenarios. This objective necessitates certain design decisions, and to this end we propose a robust, scalable framework that learns compositional functions and scores relative phrasal compositionality. We make three contributions: first, a novel way to learn compositional functions for part-of-speech pairs that uses supervision from an automatically-extracted list of paraphrases (§ SECREF3 ). Second, a context-dependent scoring model that scores the relative compositionality of a phrase BIBREF12 by computing the likelihood of its context given its paraphrase-learned representation (§ SECREF4 ). And third, an evaluation of the impact of compositionality knowledge in an end-to-end MT setup. Our experiments (§ SECREF5 ) reveal that using supervision from automatically extracted paraphrases produces compositional functions with equivalent performance to previous approaches that have relied on hand-annotated training data. Furthermore, compositionality features consistently improve the translations produced by a strong English–Spanish translation system.
section_name: Parametric Composition Functions
paragraphs: We formalize composition as a function INLINEFORM0 that maps INLINEFORM1 -dimensional vector representations of phrase constituents INLINEFORM2 to an INLINEFORM3 -dimensional vector representation of the phrase, i.e., the composed representation. A phrase is defined as any contiguous sequence of words of length 2 or greater, and does not have to adhere to constituents in a phrase structure grammar. This definition is in line with our MT application and ignores “gappy” noncontiguous phrases, but this pragmatic choice does exclude many verb-object relations BIBREF13 . We assume the existence of word-level vector representations for every word in our vocabulary of size INLINEFORM4 . Compositionality is modeled as a bilinear map, and two classes of linear models with different levels of parametrization are proposed. Unlike previous work BIBREF6 , BIBREF7 , BIBREF14 where the functions are word-specific, our compositional functions operate on part-of-speech (POS) tag pairs, which facilitates learning by drastically reducing the number of parameters, and only requires a shallow syntactic parse of the input.
section_name: Concatenation Models
paragraphs: Our first class of models is a generalization of the additive models introduced in Mitchell2008: DISPLAYFORM0 
 where the notation INLINEFORM0 represents a vertical (row-wise) concatenation of two vectors; namely, the concatenation that results in a INLINEFORM1 -sized vector. In addition to the INLINEFORM2 parameters for the word vector representations that are provided a priori, this model introduces INLINEFORM3 parameters, where INLINEFORM4 is the number of POS-tag pairs we consider.
Mitchell2008 significantly simplify parameter estimation by assuming a certain structure for the parameter matrix INLINEFORM0 , which is necessary given the limited human-annotated data they use. For example, by assuming a block-diagonal structure, we get a scaled element-wise addition model INLINEFORM1 . While not strictly in this category due to the non-linearities involved, neural network-based compositional models BIBREF7 , BIBREF15 can be viewed as concatenation models, although the order of concatenation and matrix multiplication is switched. However, these models introduce more than INLINEFORM2 parameters.
section_name: Tensor Models
paragraphs: The second class of models leverages pairwise multiplicative interactions between the components of the two word vectors: DISPLAYFORM0 
 where INLINEFORM0 corresponds to a tensor contraction along the INLINEFORM1 mode of the tensor INLINEFORM2 . In this case, we first compute a contraction (tensor-vector product) between INLINEFORM3 and INLINEFORM4 along INLINEFORM5 's third mode, corresponding to interactions with the second word vector of a two-word phrase and resulting in a matrix, which is then multiplied along its second mode (corresponding to traditional matrix multiplication on the right) by INLINEFORM6 . The final result is an INLINEFORM7 vector. This model introduces INLINEFORM8 parameters.
Tensor models are a generalization of the element-wise multiplicative model BIBREF16 , which permits non-zero values only on the tensor diagonal. Operating at the vocabulary level, the model of Baroni2010 has interesting parallels to our tensor model. They focus on adjective–noun relationships and learn a specific matrix for every adjective in their dataset; in our case, the specific matrix for each adjective has a particular form, namely that it can be factorized into the product of a tensor and a vector; the tensor corresponds to the actual adjective–noun combiner function, and the vector corresponds to specific lexical information that the adjective carries. This concept generalizes to other POS pairs: for example, multiplying the tensor that represents determiner-noun combinations along the second mode with the vector for “the” results in a matrix that represents the semantic operation of definiteness. Learning these parameters jointly is statistically more efficient than separately learning versions for each word.
section_name: Longer Phrases
paragraphs: The proposed models operate on pairs of words at a time. To handle phrases of length greater than two, we greedily construct a left-branching tree of the phrase constituents that eventually dictates the application of the learned bilinear maps. For each internal tree node, we consider the POS tags of its children: if the right child is a noun, and the left child is either a noun, adjective, or determiner, then the internal node is marked as a noun, otherwise we mark it with a generic other tag. At the end of the procedure, unattached nodes (words) are attached at the highest point in the tree.
After the tree is constructed, we can compute the overall phrasal representation in a bottom-up manner, guided by the labels of leaf and internal nodes. We note that the emphasis of this work is not to compute sentence-level representations. This goal has been explored in recent research BIBREF17 , BIBREF18 , and combining our models with methods presented therein for sentence-level representations is straightforward.
section_name: Learning
paragraphs: The models described above rely on parameters INLINEFORM0 that must be learned. In this section, we argue that automatically constructed databases of paraphrases provide adequate supervision for learning notions of compositionality.
section_name: Supervision from Automatic Paraphrases
paragraphs: The Paraphrase Database BIBREF9 is a collection of ranked monolingual paraphrases that have been extracted from word-aligned parallel corpora using the bilingual pivot method BIBREF19 . The underlying assumption is that if two strings in the same language align to the same string in another language, then the strings in the original language share the same meaning. Paraphrases are ranked by their word alignment scores, and in this work we use the preselected small portion of PPDB as our training data. Although we can directly extract phrasal representations of a pre-specified list of phrases from the corpus used to compute word representations BIBREF6 , this approach is both computationally and statistically inefficient: the number of phrases increases exponentially in the length of the phrase, and correspondingly the occurrence of any individual phrase decreases exponentially. We can thus circumvent these computational and statistical issues by using monolingual paraphrases.
The training data is filtered to provide only two-to-one word paraphrase mappings, and the multiword portion of the paraphrase is subsequently POS-tagged. Table TABREF10 provides a breakdown of such paraphrases by their POS pair type. Given the lack of context when tagging, it is likely that the POS tagger yields the most probable tag for words and not the most probable tag given the (limited) context. Furthermore, even the higher quality portions of PPDB yield paraphrases of ranging quality, ranging from non-trivial mappings such as young people INLINEFORM0 youth, to redundant ones like the ceasefire INLINEFORM1 ceasefire. However, PPDB-like resources are more easily available than human-annotated resources (in multiple languages too: Ganitkevich2014), so it is imperative that methods which learn compositional functions from such sources handle noisy supervision adequately.
section_name: Parameter Estimation
paragraphs: The parameters INLINEFORM0 in Eq. EQREF4 and EQREF6 can be estimated through standard linear regression techniques in conjunction with the data presented in § SECREF3 . These methods provide a natural way to regularize INLINEFORM1 via INLINEFORM2 (ridge) or INLINEFORM3 (LASSO) regularization, which also helps handle noisy paraphrases. Parameters for the INLINEFORM4 -regularized concatenation model for select POS pairs are displayed in Fig. FIGREF12 . The heat-maps display the relative magnitude of parameters, with positive values colored blue, negative values colored red, and white cells indicating zero values. It is evident that the parameters learned from PPDB indicate a notion of linguistic headedness, namely that for particular POS pairs, the semantic information is primarily contained in the right word, but for others such as the noun–noun combination, each constituent's contribution is relatively more equal.
section_name: Measuring of Compositionality
paragraphs: The concatenation and tensor models compute an INLINEFORM0 -dimensional vector representation for a multi-word phrase by assuming the meaning of the phrase can be expressed in terms of the meaning of its constituents. This assumption holds true to varying degrees; while it clearly holds for “large amount" and breaks down for “cloud nine", it is partially valid for phrases such as “zebra crossing" or “crash course". In line with previous work, we assume a compositionality continuum BIBREF12 , but further conjecture that a phrase's level of compositionality is dependent on the specific context in which it occurs, motivating a context-based approach (§ SECREF21 ) which scores compositionality by computing the likelihoods of surrounding context words given a phrase representation. The effect of context is directly measured through a comparison with context-independent methods from prior work BIBREF20 , BIBREF21 
It is important to note that most prior work on compositionality scoring assumes access to both word and phrase vector representations (for select phrases that will be evaluated) a priori. The latter are distinct from representations that are computed from learned compositional functions as they are extracted directly from the corpus, which is an expensive procedure. Our aim is to develop compositional models that are applicable in downstream tasks, and thus assuming pre-existing phrase vectors is unreasonable. Hence for phrases, we only rely on representations computed from our learned compositional functions.
section_name: At the Type Level
paragraphs: Given vector representations for the constituent words in a phrase and the phrase itself, the idea behind the type-based model is to compute similarities between the constituent word representations and the phrasal representation, and average the similarities across the constituents. If the contexts in which a constituent word occurs, as dictated by its vector representation, are very different from the contexts of the composed phrase, as indicated by the cosine similarity between the word and phrase representations, then the phrase is likely to be non-compositional. Assuming unit-normalized word vectors INLINEFORM0 and phrase vector INLINEFORM1 computed from one of the learned models in § SECREF2 : DISPLAYFORM0 
 where INLINEFORM0 is a hyperparameter that controls the contribution of individual constituents. This model leverages the average statistics computed over the training corpora (as encapsulated in the word and phrase vectors) to detect compositionality, and is the primary way compositionality has been evaluated previously BIBREF21 , BIBREF22 . Note that for the simple additive model INLINEFORM1 with unit-normalized word vectors, INLINEFORM2 is independent of INLINEFORM3 .
section_name: At the Token Level
paragraphs: Eq. EQREF20 scores phrases for compositionality regardless of the context that these phrases occur in. However, phrases such as “big fish" or “heavy metal" may occur in both compositional and non-compositional situations, depending on the nature and topic of the texts they occur in. Here, we propose a context-driven model for compositionality detection, inspired by the skip-gram model for learning word representations BIBREF2 . The intuition is simple: if a phrase is compositional, it should be sufficiently predictive of the context words around it; otherwise, it is acting in a non-compositional manner. Thus, we would like to compute the likelihood of the context ( INLINEFORM0 ) given a phrasal representation ( INLINEFORM1 ) and normalization constant INLINEFORM2 : DISPLAYFORM0 
 As explained in Goldberg2014, the context representations are distinct from the word representations. In practice, we compute the log-likelihood averaged over the context words or the perplexity instead of the actual likelihood.
section_name: Evaluation
paragraphs: Our experiments had three aims: first, demonstrate that the compositional functions learned using paraphrase supervision compute semantically meaningful results for compositional phrases by evaluating on a phrase similarity task (§ SECREF29 ); second, verify the hypothesis that compositionality is context-dependent by comparing a type-based and token-based approach on a compound noun evaluation task (§ SECREF36 ); and third, determine if the compositionality-scoring models based on learned representations improve the translations produced by a state-of-the-art phrase-based MT system (§ SECREF38 ).
The word vectors used in all of our experiments were produced by word2vec using the skip-gram model with 20 negative samples, a context window size of 10, a minimum token count of 3, and sub-sampling of frequent words with a parameter of INLINEFORM0 . We extracted corpus statistics for word2vec using the AFP portion of the English Gigaword, which consists of 887.5 million tokens. The code used to generate the results is available at http://www.github.com/xyz, and the evaluation datasets are publicly available.
section_name: Phrasal Similarity
paragraphs: For the phrase similarity task we first compare our concatenation and tensor models learned using INLINEFORM0 and INLINEFORM1 regularization to three baselines:
[noitemsep]
add: INLINEFORM0 
mult1: INLINEFORM0 
mult2: INLINEFORM0 
Other additive models from previous work BIBREF5 , BIBREF23 , BIBREF24 that impose varying amounts of structural assumptions on the semantic interactions between word representations e.g., INLINEFORM0 or INLINEFORM1 are subsumed by our concatenation model. The regularization strength hyperparameter for INLINEFORM2 and INLINEFORM3 regularization was selected using 5-fold cross-validation on the PPDB training data.
We evaluated the phrase compositionality models on the adjective–noun and noun–noun phrase similarity tasks compiled by Mitchell2010, using the same evaluation scheme as in the original work. Spearman's INLINEFORM0 between phrasal similarities derived from our compositional functions and the human annotators (computed individually per annotator and then averaged across all annotators) was the evaluation measure.
Figure FIGREF24 presents the correlation results for the two POS pair types as a function of the dimensionality INLINEFORM0 of the representations for the concatenation models (and additive baseline) and tensor models (and multiplicative baselines). The concatenation models seem more effective than the tensor models in the adjective–noun case and give roughly the same performance on the noun–noun dataset, which is consistent with previous work that uses dense, low-dimensional representations BIBREF25 , BIBREF15 , BIBREF26 . Since the concatenation model involve fewer parameters, we use it as the compositional model of choice for subsequent experiments. The absolute results are also consistent with state-of-the-art results on this dataset BIBREF24 , BIBREF26 , indicating that paraphrases are an excellent source of information for learning compositional functions and a reasonable alternative to human-annotated training sets. For reference, the inter-annotator agreements are 0.52 for the adjective–noun evaluation and 0.51 for the noun–noun one. The unweighted additive baseline is surprisingly very strong on the noun–noun set, so we also compare against it in subsequent experiments.
section_name: Compositionality
paragraphs: To evaluate the compositionality-scoring models, we used the compound noun compositionality dataset introduced in Reddy2011. This dataset consists of 2670 annotations of 90 compound-noun phrases exhibiting varying levels compositionality, with scores ranging from 0 to 5 provided by 30 annotators. It also contains three to five example sentences of these phrases that were shown to the annotators, which we make use of in our context-dependent model. Consistent with the original work, Spearman's INLINEFORM0 is computed on the averaged compositionality score for a phrase across all the annotators that scored that phrase (which varies per phrase). For computing the compositional functions, we evaluate three of the best performing setups from § SECREF29 : the INLINEFORM1 and INLINEFORM2 -regularized concatenation models, and the simple additive baseline.
For the context-independent model, we select the hyperparameter INLINEFORM0 in Eq. EQREF20 from the values INLINEFORM1 . For the context-dependent model, we vary the context window size INLINEFORM2 by selecting from the values INLINEFORM3 . Table TABREF37 presents Spearman's INLINEFORM4 for these setups. In all cases, the context-dependent models outperform the context-independent ones, and using a relatively simple token-based model we can approximately match the performance of the Bayesian model proposed by Hermann2012. The concatenation models are also consistently better than the additive compositional model, indicating the benefit of learning the compositional parameters via PPDB.
section_name: Machine Translation
paragraphs: While any truly successful model of semantics must match human intuitions, understanding the applications of our models is likewise important. To this end, we consider the problem of machine translation, operating under the hypothesis that sentences which express their meaning non-compositionally should also translate non-compositionally.
Modern phrase-based translation systems are faced with a large number of possible segmentations of a source-language sentence during decoding, and all segmentations are considered equally likely BIBREF13 . Thus, it would be helpful to provide guidance on more likely segmentations, as dictated by the compositionality scores of the phrases extracted from a sentence, to the decoder. A low compositionality score would ideally force the decoder to consider the entire phrase as a translation unit, due to its unique semantic characteristics. Correspondingly, a high score informs the decoder that it is safe to rely on word-level translations of the phrasal constituents. Thus, if we reveal to the translation system that a phrase is non-compositional, it should be able to learn that translation decisions which translate it as a unit are to be favored, leading to better translations.
To test this hypothesis, we built an English-Spanish MT system using the cdec decoder BIBREF27 for the entire training pipeline (word alignments, phrase extraction, feature weight tuning, and decoding). Corpora from the WMT 2011 evaluation was used to build the translation and language models, and for tuning (on news-test2010) and evaluation (on news-test2011), with scoring done using BLEU BIBREF28 . The baseline is a hierarchical phrase-based system BIBREF29 with a 4-gram language model, with feature weights tuned using MIRA BIBREF30 . For features, each translation rule is decorated with two lexical and phrasal features corresponding to the forward INLINEFORM0 and backward INLINEFORM1 conditional log frequencies, along with the log joint frequency INLINEFORM2 , the log frequency of the source phrase INLINEFORM3 , and whether the phrase pair or the source phrase is a singleton. Weights for the language model, glue rule, and word penalty are also tuned. This setup (Baseline) achieves scores en par with the published WMT results.
We added the compositionality score as an additional feature, and also added two binary-valued features: the first indicates if the given translation rule has not been decorated with a compositionality score (either because it consists of non-terminals only or the lexical items in the translation rule are unigrams), and correspondingly the second feature indicates if the translation rule has been scored. Therefore, an appropriate additional baseline would be to mark translation rules with these indicator functions but without the scores, akin to identifying rules with phrases in them (Baseline + SegOn).
Table TABREF40 presents the results of the MT evaluation, comparing the baselines to the best-performing context-independent and dependent scoring models from § SECREF36 . The scores have been averaged over three tuning runs with standard deviation in parentheses; bold results on the test set are statistically significant ( INLINEFORM0 ) with respect to the baseline. While knowledge of relative compositionality consistently helps, the improvements using the context-dependent scoring models, especially with the INLINEFORM1 concatenation model, are noticeably better.
section_name: Related Work
paragraphs: There has been a large amount of work on compositional models that operate on vector representations of words. With some exceptions BIBREF16 , BIBREF5 , all of these approaches are lexicalized i.e., parameters (generally in the form of vectors, matrices, or tensors) for specific words are learned, which works well for frequently occurring words but fails when dealing with compositions of arbitrary word sequences containing infrequent words. The functions are either learned with a neural network architecture BIBREF7 or as a linear regression BIBREF6 ; the latter require phrase representations extracted directly from the corpus for supervision, which can be computationally expensive and statistically inefficient. In contrast, we obtain this information through many-to-one PPDB mappings. Most of these models also require additional syntactic BIBREF31 or semantic BIBREF15 , BIBREF14 resources; on the other hand, our proposed approach only requires a shallow syntactic parse (POS tags). Recent efforts to make these models more practical BIBREF32 attempt to reduce their statistically complex and overly-parametrized nature, but with the exception of Zanzotto2010, who propose a way to extract compositional function training examples from a dictionary, these models generally require human-annotated data to work.
Most models that score the relative (non-) compositionality of phrases do so in a context-independent manner. A central idea is to replace phrase constituents with semantically-related words and compute the similarity of the new phrase to the original BIBREF22 , BIBREF33 or make use of a variety of lexical association measures BIBREF10 , BIBREF34 . Sporleder2009 however, do make use of context in a token-based approach, where the context in which a phrase occurs as well as the phrase itself is modeled as a lexical chain, and the cohesion of the chain is measured as an indicator of a phrase's compositionality. Cohesion is computed using a web search engine-based measure, whereas we use a probabilistic model of context given a phrase representation. Hermann2012 propose a Bayesian generative model that is also context-based, but learning and inference is done through a relatively expensive Gibbs sampling scheme.
In the context of MT, Zhang2008b present a Bayesian model that learns non-compositional phrases from a synchronous parse tree of a sentence pair. However, the primary aim of their work is phrase extraction for MT, and the non-compositional constraints are only applied to make the space of phrase pairs more tractable when bootstrapping their phrasal parser from their word-based parser. In contrast, we score every phrase that is extracted with the standard phrase extraction heuristics BIBREF29 , allowing the decoder to make the final decision on the impact of compositionality scores in translation. Thus, our work is more similar to Xiong2010, who propose maximum entropy classifiers that mark positions between words in a sentence as being a phrase boundary or not, and integrate these scores as additional features in an MT system.
section_name: Conclusion
paragraphs: In this work, we presented two new sources of information for compositionality modeling and scoring, paraphrase information and context. For modeling, we showed that the paraphrase-learned compositional representations performs as well on a phrase similarity task as the average human annotator. For scoring, the importance of context was shown through the comparison of context-independent and dependent models. Improvements by the context-dependent model on an extrinsic machine translation task corroborate the utility of these additional knowledge sources. We hope that this work encourages further research in making compositional semantic approaches applicable in downstream tasks.

Figures and tables:
caption: Table 1: Number of paraphrase examples per POS pair type out of the two-to-one word paraphrases in the SMALL version of PPDB (using the Penn Treebank tag-set). We distinguish between the five most common POS pair types, and group the remaining pairs into the generic other category.
caption: Figure 1: Parameter heat-maps for specific POS pair compositional functions. Positive values are blue, negative values red, and zero values are white. Certain phrasal relationships (e.g., DT-NN and VB-VB) exhibit headedness.
caption: Figure 2: Spearman’s ρ correlation with respect to human judgments for the adjective–noun and noun– noun phrase similarity tasks. Dashed lines correspond to tensor models (and baselines), and solid lines are concatenation models (and additive baseline).
caption: Table 2: Correlation between model judgments on phrase compositionality and human judgments, measured by Spearman’s ρ. Context-dependent (token-based) and concatenation models do better.
caption: Table 3: MT results. Bold results are statistically significant, and our best context-dependent setup is 0.4 BLEU points better than the baseline.

Questions and answers:
question: Do they study numerical properties of their obtained vectors (such as orthogonality)?
answers: answer: unanswerable: no
yes_no: no
answer: unanswerable: no
yes_no: no
question: How do they score phrasal compositionality?
answers: answer: unanswerable: no
extractive_spans: Spearman's INLINEFORM0 between phrasal similarities derived from our compositional functions and the human annotators
question: Which translation systems do they compare against?
answers: answer: unanswerable: no
extractive_spans: hierarchical phrase-based system BIBREF29
appropriate additional baseline would be to mark translation rules with these indicator functions but without the scores, akin to identifying rules with phrases in them (Baseline + SegOn)
answer: unanswerable: no
extractive_spans:  English-Spanish MT system