Venue: NAACL 2019

# Evaluating Composition Models for Verb Phrase Elliptical Sentence Embeddings

Gijs Wijnholds

Queen Mary University of London

g.j.wijnholds@qmul.ac.uk &Mehrnoosh Sadrzadeh

Queen Mary University of London

mehrnoosh.sadrzadeh@qmul.ac.uk

###### Abstract

Ellipsis is a natural language phenomenon where part of a sentence is missing and its information must be recovered from its surrounding context, as in "Cats chase dogs and so do foxes.". Formal semantics has different methods for resolving ellipsis and recovering the missing information, but the problem has not been considered for distributional semantics, where words have vector embeddings and combinations thereof provide embeddings for sentences. In elliptical sentences these combinations go beyond linear as copying of elided information is necessary. In this paper, we develop different models for embedding VP-elliptical sentences. We extend existing verb disambiguation and sentence similarity datasets to ones containing elliptical phrases and evaluate our models on these datasets for a variety of non-linear combinations and their linear counterparts. We compare results of these compositional models to state of the art holistic sentence encoders. Our results show that non-linear addition and a non-linear tensor-based composition outperform the naive non-compositional baselines and the linear models, and that sentence encoders perform well on sentence similarity, but not on verb disambiguation.

## 1 Introduction

Compositional distributional semantics has so far relied on a tight connection between syntactic and semantic resources. Based on the assembly principle of compositionality, these models assign a sentence vector by applying a linear map to the individual word embeddings therein. The meaning of "cats chase dogs" is as follows in (1) additive, (2) multiplicative, and (3) tensor-based models:

\begin{tabular}{l l}  & **\(\overrightarrow{catts}+\overrightarrow{chase}+\overrightarrow{dogs}\)** \\   & **\(\overrightarrow{catts}\odot\overrightarrow{chase\in\mathbb{O}}\overrightarrow{ dogs}\)** \\   & **\(\overrightarrow{catts}^{\top}\times(\overleftarrow{chase}\times\overrightarrow{ dogs})\)** \\ \end{tabular} Some linguistic phenomena, however, rely on copying resources while computing meaning; canonical examples thereof are anaphora and ellipsis, exemplified below:

\begin{tabular}{l l}  & **\(\quad\)** & Cats clean themselves. \\   & **\(\quad\)** & Cats chase dogs, children do too. \\ \end{tabular} More complex examples involve a structural ambiguity such as the following:

\begin{tabular}{l l}  & **\(\quad\)** & **\(\quad\)** \\ \end{tabular} These lend themselves to a strict (dogs chase the cat's tail) and a sloppy reading (dogs chase their own tail). In these examples, the meaning of at least one part of the sentence is used twice, e.g. the subject in a, the verb phrase "chase dogs" in b. Such cases can often be extended to a situation in which a meaning is used more than twice, e.g. in "Cats chase their tail, dogs too, and so do foxes".

In order to develop distributional semantics for such sentences while respecting the principle of compositionality, one has a choice between a linear or a non-linear composition of resources. In the linear case, no information is copied, resulting in vector embeddings such as the following one (when only considering content words):

\[\overrightarrow{catts}+\overrightarrow{chase}+\overrightarrow{dogs}+ \overrightarrow{children}\]

In the non-linear case, the necessary resources are copied to resolve the ellipsis, resulting in vectors embeddings such as:

\[\overrightarrow{catts}+\overrightarrow{chase}+\overrightarrow{dogs}+ \overrightarrow{children}+\overrightarrow{chase}+\overrightarrow{dogs}\]

One has the same choice when dealing with multiplicative and tensor-based models. The question is which of these composition frameworks, i.e. linear versus non-linear, provides a better choice for embedding elliptical sentences. To our knowledge, this has remained an open question: although some theoretical work has been done to

model verb phrase ellipsis in compositional distributional semantics (Wijnholds and Sadrzadeh, 2018), none of the existing datasets or evaluation methods for distributional semantics focus on elliptical phenomena.

In this paper, we provide some answers. Our starting point is the lambda logical forms of sentences, e.g. those produced by the approach of Dalrymple et al. (1991), which uses a higher order unification algorithm to resolve ellipsis. We apply to these the lambdas-to-vectors mapping of Muskens and Sadrzadeh (2016, 2017) to homomorphically map the lambda terms into concrete vector embeddings resulting from a multitude of composition operators, such as addition, multiplication, and tensor-based. We work with four vector spaces (count-based, Word2Vec, GloVe, FastText) and three different verb embeddings, and contrast our compositional models with state of the art holistic sentence encoders.

We evaluate the sentence embeddings by using them in a verb disambiguation and in a sentence similarity task, created by extending previous SVO tasks from Grefenstette and Sadrzadeh (2011a) and Kartsaklis and Sadrzadeh (2013) to an elliptical setting, and obtaining new human judgements using the Amazon Mechanical Turk crowdsourcing tool. Our experiments show that in both tasks, the models that use a non-linear form of composition perform better than the models whose composition framework is linear, suggesting that resolving ellipsis contributes to the quality of the sentence embedding.

## 2 Background

**Single-Word Embeddings:** Distributional semantics on the word level relies on the embedding of word meaning in a vectorial form: by taking context words as the basis of a vector space one computes the vector components of each word by considering its distribution among corpus data. Then a similarity measure is defined on the vector space via the cosine similarity. In a count-based model, the context is taken to be a linear window and the corpus is traversed to collect raw co-occurrence counts. Then, a weighting scheme is applied to smooth the raw frequencies in the meaning representation. More discussion on count-based vector space models can be found in Turney and Pantel (2010), and a systematic study of the parameters of count-based word embeddings is given by Kiela and Clark (2014).

With the rise of deep learning techniques, much attention has been given to neural word embeddings Mikolov et al. (2013); Pennington et al. (2014); Bojanowski et al. (2017), which try to predict rather than observe, the context of a word by optimising an objective function based on the probability of observing a context.

**Compositional Models:** The key idea of compositional models is that the meaning of elementary constituents can be combined in a structured way to obtain a representation for larger phrases. In a distributional setting, having a compositional operator is imperative: a data-driven model would not be adequate given the sparsity of full sentences in a corpus. Moreover, it is not clear that sentences follow the distributional hypothesis.

Concrete composition operators can roughly be classified as simple and tensor-based. Simple models add or multiply the word vectors to obtain a sentence vector. The work of Mitchell and Lapata (2010) experiments with these models. Tensor-based models differ in that they represent complex words as vectors of a higher order: Baroni and Zamparelli (2010) represents adjectives as matrices which, applied to a word vector produce a vector representation of the compound adjective-noun combination. The account of Coecke et al. (2010, 2013); Clark (2015) generalises this to higher-order tensors, e.g. cubes for transitive verbs and hypercubes for distransitive verbs. The benefit of a type-driven approach over the simple models is that they respect the grammatical structure of sentences: the meaning of "man bites dog" is distinct from that of "dog bites man" whereas in an additive/multiplicative model they would be identical. The trade-off is that the tensors themselves have to be learnt; where Baroni and Zamperelli (2010) apply regression learning to learn the content of adjective matrices, for transitive verbs there have been several approaches using multistep regression learning Grefenstette et al. (2013), relational learning Grefenstette and Sadrzadeh (2011a), or a combination of co-occurrence information with machine learning techniques Polajnar et al. (2014,b); Fried et al. (2015). A comparative study between count-based and neural embeddings in a compositional setting was carried out by Milajevs et al. (2014).

Neural composition turns the problem of compositionality around by learning the composition 

operator instead of predicting the result. Examples are Skip-Thought Vectors (Kiros et al., 2015), the Distributed Bag of Words model Le and Mikolov (2014), InferSent Conneau et al. (2017), and Universal Sentence Encoder Cer et al. (2018).

**Ellipsis, Formally:** There exists many formal approaches to ellipsis and anaphora in the literature. These have generally taken either a syntactic or a semantic form1. Examples of the syntactic approaches are in the work of Hendriks and Dekker (1995); Morrill and Valentin (2015); Jager (2006); Kubota and Levine (2017); these use directional extensions of categorial grammars that allow for the syntactic types at the site of ellipsis be unified with copies of the types at the antecedent of the elliptical phrase. Another approach deletes the syntactic structure at the ellipsis site and reconstruct it by copying across the antecedent structure (Fiengo and May, 1994; Merchant, 2004).

Footnote 1: Although pragmatics approaches exist (Merchant, 2010), we focus here on syntactic and semantic approaches.

Semantic approaches Dalrymple et al. (1991); Szabolcsi (1987); Pulman (1997) assume that ellipsis involves underspecification of content and resolve this by producing a predicate via a suitable abstraction from the antecedent. For instance, the elliptical phrase (b) "Cats chase dogs, children do too", will take an initial logical form \((b_{1})\); a resolution step \((b_{2})\) provides it with the lambda term in \((b_{3})\), which constitutes its final semantic form:

\[(b_{1}) \quad chase(cats,dogs)\wedge P(children)\] \[(b_{2}) \quad P=\lambda x.chase(x,dogs)\] \[(b_{3}) \quad(b_{1})\leadsto_{\beta}chase(cats,dogs)\] \[\quad\wedge chase(children,dogs)\]

The ambiguous example (d) "Cats chase their tails, dogs too" is treated similarly, but can now obtain its respective strict and sloppy readings by producing predicates \((d_{1})\) and \((d_{2})\) below:

\[(d_{2}) \quad P=\lambda x.chase(x,tail(cats))\] \[(d_{3}) \quad P=\lambda x.chase(x,tail(x))\]

Mixed syntactic/semantic approaches have also been proposed to cover wider ranges of phenomena; see Kempson et al. (2015) for an overview.

The only existing work attempting to join ellipsis analysis with vector embeddings is the proposal of Kartsaklis et al. (2016), which is preliminary work and gives unwanted results2. Below, we develop a new such approach.

Footnote 2: The meaning of “Bill brought apples and John pears” coincides with that of “Bill and John brought apples and pears”.

## 3 Embeddings for Elliptical Phrases

Vectors and their basic operations can be _emulated_ using a lambda calculus with constants for the relevant operations, as shown in Muskens and Sadrzadeh (2016). They assume a type \(I\) (a finite index set) and \(R\) (modelling the real numbers) and model any vector as a term of type \(V:=IR\); that is, as a function from indices to real numbers. Matrices can then be represented by types \(M:=IIR\) and in general a tensor of rank \(n\) will have type \(T^{n}:=I_{1}...I_{n}R\). The standard operations like scalar multiplication, addition, element wise multiplication and tensor contraction can be modelled with lambda terms as follows:

\begin{tabular}{l l l l}   \(\cdot\) & \(:=\) & \(\lambda rvi.r\cdot v_{i}\) & \(:RVV\) \\  & \(+\) & \(:=\) & \(\lambda vwi.v_{i}+w_{i}\) & \(:VVV\) \\  & \(\odot\) & \(:=\) & \(\lambda vwi.v_{i}\cdot w_{i}\) & \(:VVV\) \\  & \(\times_{1}\) & \(:=\) & \(\lambda mvii.\sum\limits_{j}m_{ij}\cdot v_{j}\) & \(:MVV\) \\  & \(\times_{2}\) & \(:=\) & \(\lambda cvijk.\sum_{k}cijk\cdot v_{k}\) & \(:T^{3}VM\) \\ \end{tabular}

The first three definitions above extend the arithmetic operations of addition and multiplication on real numbers in \(R\) to lists of numbers in \(IR\) and define corresponding definitions on vectors, and so \(\odot\) defines the pointwise multiplication of two vectors. The operation \(\times_{1}\) defines matrix multiplication; \(\times_{2}\) defines the tensor contraction between a cube \(c\) (in \(I^{3}R\)) and a list of numbers \(v\).

The vector semantics of a lambda term \(m\) is computed by taking a homomorphic image over the set of its constants \(c\). This image is computed compositionally from the vector or tensor embeddings of the constants \(c\) of \(m\) via their homomorphic images \(\mathcal{H}(c)\), whose types are denoted by \(\mathcal{T}(c)\).

Examples of these are given in Table 1 for a

\begin{table}
\begin{tabular}{l l l} \hline \(c\) & \(\mathcal{H}(c)\) & \(\mathcal{T}(c)\) \\ \hline cn & **cn** & \(V\) \\ adj & \(\lambda v.\textbf{adj}\times_{1}v)\) & \(VV\) \\ adv & \(\lambda v.\textbf{(adv}\times_{1}v)\) & \(VV\) \\ itv & \(\lambda v.\textbf{(itv}\times_{1}v)\) & \(VV\) \\ tv & \(\lambda uv.\textbf{(tv}\times_{2}v)\times_{1}u\) & \(VVV\) \\ coord & \(\lambda P.XQ.P\nabla Q\) & \(VVV\) \\ quant & \(\lambda eZ.Z(\textbf{quant}\times_{1}v)\) & \(V(VVV)V\) \\ \hline \end{tabular}
\end{table}
Table 1: Lambda Vector look up table for a tensor-based composition model. cn: a common noun, adj: adjective, adv: adverb, itv: intransitive verb, tv: transitive verb, coord: coordinator, quant: generalised quantifier; \(P,Q\) are variables of type \(V\), and so are \(v,u\), \(Z\) is a variable of type VV, \(\nabla\) is either \(\odot\) or \(+\).

 

tensor-based composition model, where the boldface \({\bf c}\) denotes the vector/tensor embedding of \(c\).

Using this table, we obtain homomorphic images of any lambda term over the constants. For instance, the lambda term of our exemplary resolved ellipsis phrase \((b_{3})\)\(\textit{chase}(\textit{cats},\textit{dogs})\wedge\textit{chase}(\textit{children}, \textit{dogs})\) is given the following semantic, obtained by computing \(\mathcal{H}(b_{3})\):

\[((\textbf{chase}\times_{2}\textbf{dogs})\times_{1}\textbf{ cats})\nabla\] \[((\textbf{chase}\times_{2}\textbf{dogs})\times_{1}\textbf{ children})\]

The constituents of the \(\mathcal{H}(c)\) entries of Table 1 are only exemplary. Many other interpretations are possible. For instance, taking vector embeddings for all words and replacing all tensor contractions and \(\nabla\) by \(+\) defines a purely additive model. The concrete models for transitive sentences that were evaluated by Milajevs et al. (2014) can all be derived by varying the \(H(c)\) entries. Below are the sentences obtained by using the Copy Object (CO), Frobenius Additive (FA), Frobenius Multiplicative (FM) and Frobenius Outer (FO) instantiations of the verb, respectively:

\[\begin{array}{ll}\textbf{CO}:&\lambda os.o\odot(\textbf{verb}\times^{\top} s)\\ \textbf{FA}:&\lambda os.s\odot(\textbf{verb}\times o)+o\odot(\textbf{verb} \times^{\top}s)\\ \textbf{FM}:&\lambda os.s\odot(\textbf{verb}\times o)\odot o\odot(\textbf{ verb}\times^{\top}s)\\ \textbf{FO}:&\lambda os.s\odot(\textbf{verb}\times o)\otimes o\odot(\textbf{ verb}\times^{\top}s)\end{array}\]

The vector semantics of the extensions of transitive sentences with VP elliptical phrases are obtained by taking each of the above as the semantics of each conjunct of the lambda logical form and interpreting the conjunction operation of \(\land\) as either sum or multiplication.

## 4 Experimental Evaluation

For the evaluation of the model(s) in the previous section, we built two new datasets and experimented with count based and neural vector spaces, and sentence encoders.3.

Footnote 3: All models, the new datasets, and evaluation code are available at github.com/gijswijnholds/compdisteval-ellipsis

### Building new datasets

In order to experiment with ellipsis, we extended the verb disambiguation dataset of Grefenstette and Sadrzadeh (2011a) and the transitive sentence similarity dataset of Kartsaklis and Sadrzadeh (2013), henceforth GS2011 and KS2013.

#### 4.1.1 Gs2011

The GS2011 verb disambiguation dataset contains 10 verbs, each with two possible interpretations. For each verb \(v\) and its two interpretations \(v_{1}\) and \(v_{2}\), the dataset contains human similarity judgments for 10 subject-object combinations. For instance, for the verb _meet_ - ambiguous between _visit_ and _satisfy_ - the dataset contains the pairs \(\langle\textit{system meet requirements, system satisfy requirements}\rangle\) and \(\langle\textit{system meet requirements, system visit requirements}\rangle\). The more likely interpretation is marked as HIGH whereas the unlikely interpretation is marked LOW.

We extended this dataset as follows: for each combination of a verb triple \((v,v_{1},v_{2})\) and a subject-object pair \((s,o)\), where \(\langle\textit{s v o, s v}_{1}~{}o\rangle\) is expected to have LOW similarity in the dataset and \(\langle\textit{s v o, s v z}~{}o\rangle\) is thus expected to have HIGH similarity, we selected a new subject \(s^{*}\) from the list of most frequent subjects for the verb \(v_{2}\) such that it was significantly more frequent for \(v_{2}\) than for \(v_{1}\)4. By doing so we strengthened the disambiguating effect of the context for each verb. The subject was selected such that the resulting elliptical phrase pairs made sense. For each combination and new subject considered, we added the two sentence pairs in the elliptical form

Footnote 4: As found in the combined ukWaC+WackyPedia corpus.

\[\begin{array}{l}\left<s\;v\;o\;and\;s^{*}\;does\;too,\;s\;v_{1}\;o\;and\;s^{*} \;does\;too\right>\\ \left<s\;v\;o\;and\;s^{*}\;does\;too,\;s\;v_{2}\;o\;and\;s^{*}\;does\;too\right> \end{array}\]

For example, for the verb triple (_draw, depict, attract_), and original sentence pairs

\[\begin{array}{l}\left<man\;draw\;word,\;man\;depic\;sword\;\right>\\ \left<man\;draw\;sword,\;man\;attract\;sword\;\right>\end{array}\]

we selected the new subject _artist_ and added two pairs, comparing _man draw sword and artist does too_ with

_man deplet sword and artist does too_

_man attract sword and artist does too_

We selected two new subjects for each combination, and in this way we obtained a dataset of roughly 400 entries. New human judgments were collected through Amazon Mechanical Turk, by prepending _the_ to each noun and putting the phrase in the past tense. As with the original dataset, participants were asked to judge the similarity between sentence pairs using a discrete number between 1 and 7; 1 for highly dissimilar, 7 for highly similar. By inserting gold standard pairs of identical sentences we checked if participants were 

trustworthy. We collected 25 judgments per sentence pair but excluded participants that annotated less than 20 entries of the total dataset. We ended up with 55 different participants who ranked more than 20 entries of the total dataset, to give a final amount of ca. 9200 annotations. As an example, the verb _show_ was a very hard case to disambiguate in the GS2011 dataset: _child show sign_ had an average score of 2.5 with both _child picture sign_ and _child express sign_. In the new dataset, with the extra subject _patient_, it got much clearer that the verb had to be interpreted as _express_ with an average score of 5.869, versus 4.875 for _picture_.

#### ks2013

The KS2013 sentence similarity dataset contains 108 transitive sentence pairs annotated with human similarity judgments. As opposed to the GS2011 dataset, subjects and objects of each sentence pair are not the same, so several different contexts get compared to one another. In this sense, the KS2013 dataset aims to investigate the role of content of individual words versus the role of composition, as the similarity of sentences might be predictable from the contribution of individual words rather than the specific way of composing them.

We extend this dataset to cover VP ellipsis by following a similar procedure as for GS2011. For each transitive sentence of the form \(s~{}v~{}o\) in the dataset, we selected a new subject \(s^{*}\) from a list of most frequent subjects of the verb5 and built elliptical entries \(s~{}v~{}o\)_and_\(s^{*}\)_does too_ in such a way that the meaning of the original transitive sentence got changed as little as possible and that the resulting elliptical phrase made sense. We then considered every transitive sentence pair in the dataset and added the new respective subjects to both sentences. For example, for the pair

Footnote 5: Again taken from the ukWaC+Wackypedia corpus.

\[\langle\textit{school encourage child, employee leave company}\rangle\]

we selected _parent_ and _student_ to get the new pair

\[\langle\textit{school encourage child and parent does too,}\] \[\textit{employee leave company and student does too}\rangle\]

We chose two subjects for every original sentence, generating four possibilities for each sentence pair, and a new dataset of 432 entries. This dataset was also annotated using Amazon Mechanical Turk, after putting each verb in the past tense and prepending _the_ to each noun in the dataset. Gold standard pairs of identical sentences were inserted to validate trustworthiness of participants. The final dataset contains ca. 9800 annotations by 42 different participants.

### Vector Spaces

To provide a comprehensive study with robust results, we used four vector spaces: a count based vector space, and newly trained Word2Vec, GloVe, and FastText spaces, as detailed below.

**Count-Based:** We used the combined ukWaC and Wackypedia corpora6 to extract raw co-occurrence counts, using as a basis the 2000 most frequently occurring tokens (after excluding the 50 most frequent ones). When extracting counts, we disregarded a list of stopwords that do not contribute to the content of the vectors. We used a context window of 5 around the focus word, and PPMI as weighting scheme. These settings were use in the original KS2013 dataset Kartsaklis and Sadrzadeh (2013).

Footnote 6: wacky.ssmlit.unibo.it

**Word2Vec:** The Word2Vec embeddings we used were trained with the continuous bag of words model of Mikolov et al. (2013) (CBOW). We trained this model on the combined and lemmatised ukWaC and Wackypedia corpora, using the implementation for Python available in the gensim package7, with a minimum word frequency of 50, a window of 5, dimensionality 300, and 5 training iterations.

Footnote 7: radimrehurek.com/gensim

**GloVe:** The GloVe model Pennington et al. (2014) considers the _ratio_ of co-occurrence probabilities by minimising the least-squares objective between the dot product of two word embeddings and the log-probability of the words' co-occurrence. We trained a GloVe space on the combined and lemmatised ukWaC and Wackypedia corpora, using the code provided by the original authors8. Similar to the Word2Vec settings above, we trained 300 dimensional vectors with a minimum word frequency of 50 and a window of 5, but we trained with 15 iterations.

Footnote 8: nlp.stanford.edu/projects/glove

**FastText:** The FastText vectors are like Word2Vec, except the word vector takes into account subword information: words are represented as \(n\)-grams, for which vectors are trained. The final word vector will then be the sum of its constituent \(n\)-gram vectors Bojanowski et al. (2017). We trained a FastText space 

as the Word2Vec space (CBOW, minimum word frequency 50, dimensions 300, window 5, with 5 iterations), again using gensim.

#### 4.2.1 Verb Matrices

In order to work with tensor-based models we had to represent verbs as matrices rather than as vectors. We generated verb tensors using two methods that have been used previously in the literature [68, 7].

**Relational:** For each verb, its corresponding matrix is obtained by summing over the tensor product of the respective subject and object vectors of the verb (subjects and objects collected from the corpus):

\[\overrightarrow{verb}=\sum_{i}subj_{i}\otimes obj_{i}\]

**Kronecker:** For each verb, its corresponding matrix is obtained by taking the tensor product of the verb vector with itself:

\[\widetilde{verb}=\overrightarrow{verb}\otimes\overrightarrow{verb}\]

In the case of the count-based space, we trained verb matrices of dimensions \(2000\times 2000\), for the neural word embeddings the matrices had dimensions \(300\times 300\). We also experimented with the skip-gram extension of Maillard and Clark (2015) and the plausibility model of Polajnar et al. (2014a) but excluded the results because the obtained verb matrices were far below par.

### Concrete Models

For the experiments, we had two main goals in mind: primarily, we wanted to verify that resolving ellipsis contributes to the performance of a compositional model. For this purpose we experimented with non-linear models, i.e. models that resolve the ellipsis (and thus use the verb and object resources twice) versus linear models, which do not resolve the ellipsis (and thus only use the verb and object once). Our second goal was to investigate whether amongst the models that resolve the ellipsis, the ones that did so in a tensor-based way, i.e. using tensors instead of vectors to represent the verbs, performed better than additive and multiplicative models, and how these compare to holistic sentence encoders. Hence, we considered three classes of models: linear vector models, nonlinear vector models and tensor-based models.

**Linear Vector Models:** These models use every resource exactly one, following the pattern \(\overrightarrow{w_{1}}\star\overrightarrow{w_{2}}...\star \overrightarrow{w_{n}}\) for any sequence of words \(w_{1}w_{2}....w_{n}\). For an elliptical phrase "_subj verb obj and subj* does too_" it will compute the vector

\[\overrightarrow{subj}\star\overrightarrow{verb}\star\overrightarrow{obj j}\star\overrightarrow{and}\star\overrightarrow{subj}\star\overrightarrow{does}\star\overrightarrow{tooo}\]

where \(\star\) denotes either addition or multiplication.

**Non-Linear Vector Models:** Here, the assumption is that ellipsis is resolved but models do not respect word order. The meaning of "_subj verb obj and subj* does too_" now is

\[\overrightarrow{subj}\star\overrightarrow{verb}\star\overrightarrow{ objj}\star\overrightarrow{subj}\star\overrightarrow{verb}\star\overrightarrow{obj}\]

**Tensor-Based Models:** These models all are assumed to resolve ellipsis and are based on various previous models [68, 79, 70]. Essentially, the tensor-based meaning of "_subj verb obj and subj* does too_" is

\[T(\overrightarrow{subj},\overrightarrow{verb},\overrightarrow{objj})\star T (\overrightarrow{subj^{\frac{1}{k}}},\overrightarrow{verb}, \overrightarrow{objj})\]

where \(T\) is a transitive model from [70] and \(\star\) interprets the conjunction of the two subclauses. For the verb matrix we used either the relational verb or the Kronecker verb, and for \(\star\) we tried both addition and multiplication. We did consider a model which simply adds or multiplies the second subject without duplicating the verb phrase, but it performed worse than non-linear addition and multiplication so we did not include it in this paper.

**Sentence Encoders:** To compare the mentioned compositional models with state of the art neural baselines, we carried out our experiments with a four types of holistic sentence encoders, that take arbitrary text as input and produce an embedding. To properly compare with the compositional models above, we gave three different inputs to the encoders: a baseline encoding (Base), a resolved encoding (Res), and an encoding without functional words (Abl), all as below:

**Base**: "_subj verb obj and subj* does too_"

**Res**: "_subj verb obj and subj* verb obj_"

**Abl**: "_subj verb obj subj_""

We used six concrete pretrained encoders, available online: 4800-dimensional embeddings from the Skip-Thought model9, 300-dimensional embeddings from two Doc2Vec implementations 

(Lau and Baldwin, 2016)10, 4096-dimensional embeddings from two InferSent encoders11, and 512-dimensional embeddings from Universal Sentence Encoder12.

Footnote 10: github.com/jhlau/doc2vec

Footnote 11: github.com/facebookresearch/InferSent

## 5 Results

To validate the quality of the trained word spaces, we evaluate on several standard word similarity tasks: we used Rubenstein & Goodenough (RG, 1965), WordSim353 (WS353, 2001), Miller & Charles (MC, 1991), SimLex-999 (SL99, 2015), and the MEN dataset (Bruni et al., 2012). The results are displayed in Table 2, for the spaces described in the previous section.

**Verb Disambiguation:** Table 3 shows the results of the linear, non-linear and tensor-based models for this task, compared against a baseline in which only the verb vector or verb matrix is compared.

Our first observation is that generally, the highest performing models were tensor-based. The highest found correlation score was 0.5385 in the count based space for a tensor-based model (**CO** model above, Kronecker matrix, \(\nabla=+\)), with the Frobenius Additive model giving the second best result of 0.5263 (**FA** model above, Kronecker matrix, \(\nabla=+\)). For the neural spaces, the highest performing models were mostly tensor-based as well; they were always the Frobenius Additive (**FA**) model and the Frobenius Outer (**FO**) model, using the relational tensor and addition for the coordinator, except in the case of GloVe, where the Copy Object (**CO**) model was the second best. The only exception to this observation is the GloVe space, for which the baseline Vector Only model in fact has a higher correlation than any other model on that space.

Our second observation is that the non-linear variants of the additive and multiplicative models (which resolve ellipsis but in a naive way) show an increased performance over the linear models (which do not resolve ellipsis). All of this holds for all the four vector spaces, except for the FastText space where the linear multiplicative model achieves significantly higher correlation (0.2928) than its non-linear counterpart (0.0440).

Overall, these results suggests that a logical resolving of ellipsis and further grammatical sensitivity benefits the performance of composition.

One interesting fact about our results is that the best compositional methods across the board were those that interpret the coordinator 'and' as addition; in set-theoretic semantics one interprets this coordinator as set intersection, which corresponds to multiplication rather than addition in a vectorial setting. We suggest that the feature intersection approach using multiplication leads to sparsity in the resulting vectorial representation, which then has a negative effect on the overall result. This would explain the case of FastText, since those vectors take into account subword information one would expect them to be more fine-grained and therefore conflate more of their features under multiplication.

The choice of verb matrix was mixed: for the count-based models the Kronecker matrix worked best, for the neural embeddings it was best to use

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **D2V1** & **D2V2** & **ST** & **IS1** & **IS2** & **USE** \\ \hline Base & .1448 & .2432 & -.1932 & .3471 & **.3841** & .2693 \\ Res & .2340 & .2980 & -.1720 & .3436 & .3373 & .2770 \\ Abl & .1899 & .2423 & -.1297 & .3525 & .3571 & .2402 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Spearman \(\rho\) scores for the ellipsis disambiguation experiment. D2V1: Doc2Vec1, D2V2: Doc2Vec 2, ST: Skip-Thought, IS1: InferSent 1, IS2: InferSent 2, USE: Universal Sentence Encoder.

\begin{table}
\begin{tabular}{l r r r r} \hline \hline  & **CB** & **W2V** & **GloVe** & **FT** \\ \hline Verb Only Vector & .4363 & .2406 & .4451 & .2290 \\ Verb Only Tensor & .3295 & .4376 & .3942 & .3876 \\ \hline Add. Linear & .4416 & .2728 & .3046 & .1409 \\ Mult. Linear & 3250 & -.0123 & .1821 & .2928 \\ \hline Add. Non-Linear & .4448 & .3275 & .3262 & .1399 \\ Mult. Non-Linear & .5029 & .2087 & .2446 & .0440 \\ \hline Best Tensor & **.5385** & .4621 & .3688 & .4937 \\
2nd Best Tensor & .5263 & .4544 & .3581 & .4652 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Spearman \(\rho\) scores for the ellipsis disambiguation experiment. CB: count based, W2V: Word2Vec, FT: FastText.

 

the relational matrix.

In comparison, the sentence encoder results of Table 4 show the same trend that suggests that resolving ellipsis improves the quality of the embeddings: with the exception of the two InferSent encoders, the resolved models gave a higher correlation than their linear baseline. However, none of the encoder models come near the results achieved using the compositional models. Since the verb disambiguation dataset contains pairs of sentence that only differ in the verb, the task becomes very much grammar-oriented, and so we argue that the tensor-based models work better since they explicitly emphasise syntactic structure.

**Sentence Similarity:** For the extension of the KS2013 sentence similarity dataset, the results are shown in Table 5. We again wanted to see if resolving ellipsis benefits the compositional process. This was in general true, although we observed a different pattern to the previous experiment.

In all cases, except for the FastText space, we saw that non-linear models in fact perform better than their linear counterparts. But this time the best tensor-based models only outperformed addition for the count-based space: the best models scored 0.7410 and 0.7370 (respectively for the **FO** and **FA** models above, Kronecker matrix, \(\nabla=\odot\)). Both Word2Vec and GloVe worked best with a non-linear additive model, with Word2Vec achieving the overall highest correlation score of 0.7617, and GloVe achieving 0.7103. For FastText, the highest score of 0.7408 was achieved by linear addition. What is more, the multiplicative model did not benefit from a non-linear approach in the case of GloVe (from 0.3666 to 0.2439), and the additive model had a similar decline in performance for the count-based space (from 0.7000 to 0.6808) and FastText (0.7408 to 0.7387). We can see that for the neural word embeddings the additive models work best, with all of them seeing a drop in performance for the tensor-based models.

Again, the best count-based models use the Kronecker matrix whereas the neural models benefit the most from using the relational matrix. However, this time the best count-based models used multiplication for coordination, the neural models preferring addition.

The sentence encoders worked a lot better in the similarity task, with all non-linear resolved models outperforming the baseline model, and the InferSent model even outperforming non-linear addition on a Word2Vec space. We argue this is the case for two reasons: first, the similarity dataset is more diffuse than the verb disambiguation dataset since sentence pairs now differ for every word in the sentence, giving more opportunity to exploit semantic similarity rather than syntactic similarity. Second, the embeddings from the sentence encoder are larger (4096), allowing them to effectively store more information to benefit the similarity score.

Overall we conclude again that resolving ellipsis improves the performance of composition, but this time the InferSent sentence encoder seems to work best, followed by the non-linear additive compositional model on Word2Vec, with tensor-based models only performing well in a count-based space.

## 6 Conclusion

In this paper we experimented with vector space semantics for VP ellipsis, working with a large variety of compositional models. We created two new datasets and compared the performance of several compositional methods, both linear and non-linear, across four vector spaces, and against state of the art holistic sentence encoders.

Our main conclusion is that resolving ellipsis improves performance: non-linear models almost

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & **D2V1** & **D2V2** & **ST** & **IS1** & **IS2** & **USE** \\ \hline Base & .5901 & .6188 & .5851 & .7785 & .7009 & .6463 \\ Res & .6878 & .6875 & .6039 & **.8022** & .7486 & .6791 \\ Abl & .1840 & .6599 & .4715 & .7815 & .7301 & .6397 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Spearman \(\rho\) scores for the ellipsis similarity experiment. D2V1: Doc2Vec1, D2V2: Doc2Vec 2, ST: Skip-Thought, IS1: InferSent 1, IS2: InferSent 2, USE: Universal Sentence Encoder.

\begin{table}
\begin{tabular}{l l r r r} \hline \hline  & **CB** & **W2V** & **GloVe** & **FT** \\ \hline Verb Only Vector & .4562 & .5833 & .4348 & .6513 \\ Verb Only Tensor & .3946 & .5664 & .4426 & .5337 \\ \hline Add. Linear & .7000 & .7258 & .6964 & .7408 \\ Mult. Linear & .6330 & .1302 & .3666 & .1995 \\ \hline Add. Non-Linear & .6808 & **.7617** & .7103 & .7387 \\ Mult. Non-Linear & .7237 & .3550 & .2439 & .4500 \\ \hline Best Tensor & .7410 & .7061 & .4907 & .6989 \\
2nd Best Tensor & .7370 & .6713 & .4819 & .6871 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Spearman \(\rho\) scores for the ellipsis similarity experiment.

 

always performed better than linear ones in both a verb disambiguation and a sentence similarity task. The highest performance on the verb disambiguation task was given by a grammar-driven, tensor-based model in a count-based vector space, whereas for the similarity task, the highest performance was achieved by the InferSent sentence encoder, followed by a non-linear additive model on a Word2Vec space. Although the neural word embeddings and sentence encoders were largely outperformed on the disambiguation dataset that places more emphasis on syntactic structure than on semantic similarity, they generally performed better in the sentence similarity case, where the distinction between syntactic and semantic similarity is more diffuse.

## Acknowledgments

The authors gratefully acknowledge the three anonymous reviewers for their valuable comments. Mehmoosh Sadrzadeh is grateful to the Royal Society for an International Exchange Award _IE161631 - Dynamic Vector Semantics for Lambda Calculus Models of Natural Language_ and discussion with Reinhard Muskens in this context. Gijs Wijnholds would like to express gratitude for support by a Queen Mary Principal Studentship, and the Theory group of the School of Electronic Engineering and Computer Science at Queen Mary University of London. Both authors would like to thank Ruth Kempson and Matthew Purver for many helpful discussions.

## References

* Baroni and Zamparelli (2010) Marco Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 1183-1193. Association for Computational Linguistics.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association of Computational Linguistics, 5(1):135-146.
* Bruni et al. (2012) Elia Bruni, Germna Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, Association for Computational Linguistics.
* Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
* Clark (2015) Stephen Clark. 2015. _Vector space models of lexical meaning_, chapter 16. John Wiley & Sons, Ltd.
* Coecke et al. (2013) Bob Coecke, Edward Grefenstette, and Mehrnoosh Sadrzadeh. 2013. Lambek vs. Lambek: Functor space semantics and string diagrams for Lambek calculus. Annals of Pure and Applied Logic, 164(11):1079-1100.
* Coecke et al. (2010) Bob Coecke, Mehrnoosh Sadrzadeh, and Stephen Clark. 2010. Mathematical foundations for a compositional distributional model of meaning. arXiv preprint arXiv:1003.4394.
* Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670-680.
* Dalrymple et al. (1991) Mary Dalrymple, Stuart M Shieber, and Fernando CN Pereira. 1991. Ellipsis and higher-order unification. Linguistics and Philosophy, 14(4):399-452.
* Fiengo and May (1994) Robert Fiengo and Robert May. 1994. Indices and identity. MIT Press, Cambridge, Mass.
* Finkelstein et al. (2001) Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th International Conference on World Wide Web, pages 406-414. ACM.
* Fried et al. (2015) Daniel Fried, Tamara Polajnar, and Stephen Clark. 2015. Low-Rank tensors for verbs in compositional distributional semantics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 731-736.
* Grefenstatte et al. (2013) Edward Grefenstatte, Georgiana Dinu, Yao-Zhong Zhang, Mehrnoosh Sadrzadeh, and Marco Baroni. 2013. Multi-Step regression learning for compositional distributional semantics. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013).
* Grefenstate and Sadrzadeh (2011a) Edward Grefenstate and Mehrnoosh Sadrzadeh. 2011a. Experimental support for a categorical compositional distributional model of meaning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1394-1404. Association for Computational Linguistics.
* Grefenstatte and Sadrzadeh (2011b) Edward Grefenstatte and Mehrnoosh Sadrzadeh. 2011b. Experimenting with transitive verbs in a DiscoCat. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language 

Semantics_, pages 62-66. Association for Computational Linguistics.
* Hendriks and Dekker (1995) Herman Hendriks and Paul Dekker. 1995. Links without locations. In _Proceedings of the Tenth Amsterdam Colloquium_, pages 339-358. Citeseer.
* Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. _Computational Linguistics_, 41(4):665-695.
* Jager (2006) Gerhard Jager. 2006. _Anaphora and type logical grammar_, volume 24. Trends in Logic. Springer Science & Business Media.
* Kartsaklis et al. (2016) Dimitri Kartsaklis, Matthew Purver, and Mehrnoosh Sadrzadeh. 2016. Verb phrase ellipsis using Frobenius algebras in categorical compositional distributional semantics. _DSALT Workshop, European Summer School on Logic, Language and Information_.
* Kartsaklis and Sadrzadeh (2013) Dimitri Kartsaklis and Mehronooh Sadrzadeh. 2013. Prior disambiguation of word tensors for constructing sentence vectors. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1590-1601.
* Kartsaklis and Sadrzadeh (2014) Dimitri Kartsaklis and Mehroosh Sadrzadeh. 2014. A study of entanglement in a categorical framework of natural language. In _Proceedings of the 11th Workshop on Quantum Physics and Logic (QPL)_. Kyoto Japan.
* Kartsaklis et al. (2012) Dimitri Kartsaklis, Mehronoosh Sadrzadeh, and Stephen Pulman. 2012. A unified sentence space for categorical distributional-compositional semantics: Theory and experiments. In _Proceedings of 24th International Conference on Computational Linguistics (COLING): Posters_. Mumbai, India.
* Kempson et al. (2015) Ruth Kempson, Ronnie Cann, Arash Eshghi, Eleni Gregoromichelaki, and Matthew Purver. 2015. Eliipsis. In S. Lappin and C. Fox, editors, _Handbook of Contemporary Semantic Theory_, 2nd edition, chapter 4. Wiley.
* Kiela and Clark (2014) Douwe Kiela and Stephen Clark. 2014. A systematic study of semantic vector space model parameters. In _Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC)_, pages 21-30.
* Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-Thought vectors. In _Advances in Neural Information Processing Systems_, pages 3294-3302.
* Kubota and Levine (2017) Yusuke Kubota and Robert Levine. 2017. Pseudo-gapping as pseudo-VP-ellipsis. _Linguistic Inquiry_, 48(2):213-257.
* Lau and Baldwin (2016) Jey Han Lau and Timothy Baldwin. 2016. An empirical evaluation of doc2vec with practical insights into document embedding generation. In _Proceedings of the 1st Workshop on Representation Learning for NLP_, pages 78-86.
* Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In _International Conference on Machine Learning_, pages 1188-1196.
* Maillard and Clark (2015) Jean Maillard and Stephen Clark. 2015. Learning adjective meanings with a tensor-based skip-gram model. In _Proceedings of the Nineteenth Conference on Computational Natural Language Learning_, pages 327-331.
* Merchant (2004) Jason Merchant. 2004. Fragments and ellipsis. _Linguistics and Philosophy_, 27:661-738.
* Merchant (2010) Jason Merchant. 2010. Three types of ellipsis. _Context-Dependence, Perspective and Relativity_, 6:141.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in Neural Information Processing Systems_, pages 3111-3119.
* Milajevs et al. (2014) Dmitrijs Milajevs, Dimitri Kartsaklis, Mehronoosh Sadrzadeh, and Matthew Purver. 2014. Evaluating neural word representations in tensor-based compositional settings. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 708-719.
* Miller and Charles (1991) George A Miller and Walter G Charles. 1991. Contextual correlates of semantic similarity. _Language and Cognitive Processes_, 6(1):1-28.
* Mitchell and Lapata (2010) Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. _Cognitive Science_, 34(8):1388-1429.
* Morrill and Valentin (2015) Glyn Morrill and Oriol Valentin. 2015. Computational coverage of TLG: Nonlinearity. In _Proceedings of NLCS'15. Third Workshop on Natural Language and Computer Science_, volume 32, pages 51-63. EasyChain Publications.
* Muskens and Sadrzadeh (2016) Reinhard Muskens and Mehrnoods Sadrzadeh. 2016. Context update for lambdas and vectors. In _International Conference on Logical Aspects of Computational Linguistics_, pages 247-254. Springer.
* Muskens and Sadrzadeh (2017) Reinhard Muskens and Mehrnoods Sadrzadeh. 2017. Lambdas, vectors, and word meaning in context. In _Proceedings of the 21st Amsterdam Colloquium_, pages 65-74.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543 

Tamara Polajnar, Luana Fagarasan, and Stephen Clark. 2014a. Reducing dimensions of tensors in type-driven distributional semantics. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1036-1046.
* Polajnar et al. (2014b) Tamara Polajnar, Laura Rimell, and Stephen Clark. 2014b. Using sentence plausibility to learn the semantics of transitive verbs. _arXiv preprint arXiv:1411.7942_.
* Pulman (1997) Stephen G. Pulman. 1997. Higher order unification and the interpretation of focus. _Linguistics and Philosophy_, 20(1):73-115.
* Rubenstein and Goodenough (1965) Herbert Rubenstein and John B Goodenough. 1965. Contextual correlates of synonymy. _Communications of the ACM_, 8(10):627-633.
* Szabolcsi (1987) Anna Szabolcsi. 1987. Bound variables in syntax (are there any?). _Sixth Amsterdam Colloquium Proceedings_.
* Turney and Pantel (2010) Peter D Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. _Journal of Artificial Intelligence Research_, 37:141-188.
* Wijnholds and Sadrzadeh (2018) Gijs Wijnholds and Mehrnoosh Sadrzadeh. 2018. Classical copying versus quantum entanglement in natural language: The case of VP-ellipsis. In _EPTCS Proceedings of the Workshop on Compositional Approaches for Physics, NLP, and Social Sciences_, volume 283, pages 103-119.