# Distributional Sentence Entailment Using Density Matrices

Esma Balkir, Mehrnoosh Sadrzadeh, Bob Coecke

► To cite this version:

Esma Balkir, Mehrnoosh Sadrzadeh, Bob Coecke. Distributional Sentence Entailment Using Density Matrices. 1st International Conference on Theoretical Computer Science (TTCS), Aug 2015, Tehran, Iran. pp.1-22, 10.1007/978-3-319-28678-5_1 . hal-01446260

HAL Id: hal-01446260

https://inria.hal.science/hal-01446260

Submitted on 25 Jan 2017

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Distributional Sentence Entailment Using Density Matrices

Esma Balkir<fnref n="1" />, Mehrnoosh Sadrzadeh<fnref n="1" />, and Bob Coecke<fnref n="2" />

**Abstract.** Categorical compositional distributional model of Clark, Coecke, and Sadrzadeh suggests a way to combine grammatical composition of the formal, type logical models with the corpus based, empirical word representations of distributional semantics. This paper contributes to the project by expanding the model to also capture entailment relations. This is achieved by extending the representations of words from points in meaning space to density operators, which are probability distributions on the subspaces of the space. A symmetric measure of similarity and an asymmetric measure of entailment is defined, where lexical entailment is measured using von Neumann entropy, the quantum variant of Kullback-Leibler divergence. Lexical entailment, combined with the composition map on word representations, provides a method to obtain entailment relations on the level of sentences. Truth theoretic and corpus-based examples are provided.

## 1 Introduction

The term *distributional semantics* is almost synonymous with the term *vector space models of meaning*. This is because vector spaces are natural candidates for modelling the *distributional hypothesis* and contextual similarity between words [11]. In a nutshell, this hypothesis says that words that often occur in the same contexts have similar meanings. So for instance, ‘ale’ and ‘lager’ are similar since they both often occur in the context of ‘beer’, ‘pub’, and ‘pint’. The obvious practicality of these models, however, does not guarantee that they possess the expressive power needed to model all aspects of meaning. Current distributional models mostly fall short of successfully modelling subsumption and entailment [19]. There are a number of models that use distributional similarity to enhance textual entailment [13, 4]. However, most of the work from the distributional semantics community has been focused on developing more sophisticated metrics on vector representations [20, 17, 27].

In this paper we suggest the use of density matrices instead of vector spaces as the basic distributional representations for the meanings of words. Density matrices are widely used in quantum mechanics, and are a generalization of vectors. There are several advantages to using density matrices to model meaning. Firstly, density matrices have the expressive power to represent all the information vectors can represent: they are a suitable implementation of the distributional hypothesis. They come equipped with a measure of information content, and so provide a natural way of implementing asymmetric relations between words such as hyponymy-hypernymy

<footnote n="1">¹ Queen Mary University of London</footnote>

<footnote n="2">² University of Oxford</footnote>
relations. Furthermore, they form a compact closed category. This allows the previous work of [7, 10] on obtaining representations for meanings of sentences from the meaning of words to be applicable to density matrices. The categorical map from meanings of words to the meaning of the sentence respects the order induced by the relative entropy of density matrices. This promises, given suitable representations of individual words, a method to obtain entailment relations on the level of sentences, inline with the lexical entailment of *natural logic*, e.g., see [21], rather than the traditional logical entailment of Montague semantics.

***Related Work.*** This work builds upon and relates to the literature on compositional distributional models, distributional lexical entailment, and the use of density matrices in computational linguistics and information retrieval.

There has been a recent interest in methods of composition within the distributional semantics framework. There are a number of composition methods in literature. See [14] for a survey of compositional distributional models and a discussion of their strengths and weaknesses. This work extends the work presented in [7, 10], a compositional model based on category theory. Their model was shown to outperform the competing compositional models in [15].

Research on distributional entailment has mostly been focused on lexical entailment. One notable exception is the work in [3], which uses the distributional data on adjective-noun and quantifier-noun pairs to train a classifier; the results are then utilized to detect novel noun pairs that have the same relation. There are a number of non-symmetric lexical entailment measures, e.g., see [27, 8, 17, 19], all of which rely on some variation of the *Distributional Inclusion Hypothesis*: "If *u* is semantically narrower than *v*, then a significant number of salient distributional features of *u* are also included in the feature vector of *v*" [17]. In their experiments, the authors of [12] show that while if a word *v* entails another word *w* then the characteristic features of *v* is a subset of the ones for *w*, it is not necessarily the case that the inclusion of the characteristic features *v* in *w* indicate that *v* entails *w*. One of their suggestions for increasing the prediction power of their method is to include more than one word in the features.

The work presented in [23] uses a measure based on entropy to detect hyponym-hypernym relationships in given pairs. The measure they suggest rely on the hypothesis that hypernyms are semantically more general than hyponyms, and therefore tend to occur in less informative contexts. The authors of [16] rely on a very similar idea, and use KL-divergence between the target word and the basis words to quantify the semantic content of the target word. They conclude that this method performs equally well in detecting hyponym-hypernym pairs as their baseline prediction method that only considers the overall frequency of the word in corpus. They reject the hypothesis that more general words occur in less informative contexts. Their method differs from ours in that they use relative entropy to quantify the overall information content of a word, and not to compare two target words to each other.

The work presented in [22] extend the compositional model of [7, 10] to include density matrices as we do, but use it for modeling homonymy and polysemy. Their approach is complementary to ours, and in fact, they show that it is possible to merge the two constructions. The work presented in [6] uses density matrices to model context effects in a conceptual space. In their quantum mechanics inspired model, words are
represented by mixed states and each eigenstate represents a sense of the word. Context effects are then modelled as quantum collapse. The authors in [5] use density matrices to encode dependency neighbourhoods, with the aim of modelling context effects in similarity tasks; the work presented in [26] uses density matrices to sketch out a theory of information retrieval, and connects the logic of the space and of density matrices via an order relation that makes the set of projectors in a Hilbert space into a complete lattice. They then use this order to define an entailment relation. Finally, the work presented in [25] shows that using density matrices to represent documents provides significant improvement on realistic IR tasks.

This paper is based on the MSc Thesis of the first author [2].

## 2 Background

**Definition 1.** A monoidal category is compact closed if for any object A, there are left and right dual objects, i.e. objects $A^r$ and $A^l$, and morphisms $\eta^l: I \to A \otimes A^l$, $\eta^r: I \to A^r \otimes A$, $\epsilon^l: A^l \otimes A \to I$ and $\epsilon^r: A \otimes A^r \to I$ that satisfy:

$$ (1_A \otimes \epsilon^l) \circ (\eta^l \otimes 1_A) = 1_A \quad (\epsilon^r \otimes 1_A) \circ (1_A \otimes \eta^r) = 1_A \\ (\epsilon^l \otimes 1_{A^l}) \circ (1_{A^l} \otimes \eta^l) = 1_{A^l} \quad (1_{A^r} \otimes \epsilon^r) \circ (\eta^r \otimes 1_{A^r}) = 1_{A^r} $$

Compact closed categories are used to represent correlations, and in categorical quantum mechanics they model maximally entangled states. [1] The $\eta$ and $\epsilon$ maps are useful in modeling the interactions of the different parts of a system. To see how this relates to natural language, consider a simple sentence with an object, a subject and a transitive verb. The meaning of the entire sentence is not simply an accumulation of the meanings of its individual words, but depends on how the transitive verb relates the subject and the object. The $\eta$ and $\epsilon$ maps provide the mathematical formalism to specify such interactions. The distinct left and right duals ensure that compact closed categories can take word order into account.

There is a graphical calculus used to reason about monoidal categories [9]. In the graphical language, objects are wires, and morphisms are boxes with incoming and outgoing wires of types corresponding to the input and output types of the morphism. The identity object is depicted as empty space, so a state $\psi: I \to A$ is depicted as a box with no input wire and an output wire with type A. The duals of states are called effects, and they are of type $A \to I$. Let $f: A \to B$, $g: B \to C$ and $h: C \to D$, and $1_A: A \to A$ the identity function on A. $1_A$, $f$, $f \otimes h$, $g \circ f$ are depicted as follows:

<figure><img src="image_2.png" /></figure>

The state $\psi: I \to A$, the effect $\pi: A \to I$, and the scalar $\psi \circ \pi$ are depicted as follows:
<figure><img src="image_3.png" /></figure>

The maps $\eta^l, \eta^r, \epsilon^l$ and $\epsilon^r$ take the following forms in the graphical calculus:

<figure><img src="image_4.png" /></figure>

The axioms of compact closure, referred to as the *snake identities* because of the visual form they take in the graphical calculus, are represented as follows:

<figure><img src="image_5.png" /></figure>

More generally, the reduction rules for diagrammatic calculus allow continuous deformations. One such deformation we will make use of is the *swing rule*:

<figure><img src="image_6.png" /></figure>

**Definition 2.** [18] A pregroup $(P, \leq, \cdot, 1, (-)^l, (-)^r)$ is a partially ordered monoid in which each element $a$ has both a left adjoint $a^l$ and a right adjoint $a^r$ such that $a^l a \leq 1 \leq aa^l$ and $aa^r \leq 1 \leq a^r a$.

If $a \leq b$ it is common practice to write $a \to b$ and say that $a$ reduces to $b$. This terminology is useful when pregroups are applied to natural language, where each word gets assigned a pregroup type freely generated from a set of basic elements. The sentence is deemed to be grammatical if the concatenation of the types of the words reduce to the simple type of a sentence. For example reduction for a simple transitive sentence is $n(n^r sn^l)n \to 1sn^ln \to 1s1 \to s$.

A pregroup $P$ is a concrete instance of a compact closed category. The $\eta^l, \eta^r, \epsilon^l, \epsilon^r$ maps are $\eta^l = [1 \leq p \cdot p^l], \epsilon^l = [p^l \cdot p \leq 1], \eta^r = [1 \leq p^r \cdot p], \epsilon^r = [p \cdot p^r \leq 1]$.
**FVect as a concrete compact closed category.** Finite dimensional vector spaces over the base field $\mathbb{R}$, together with linear maps form a monoidal category, referred to as **FVect**. The monoidal tensor is the usual vector space tensor and the monoidal unit is the base field $\mathbb{R}$. It is also a compact closed category where $V^l = V^r = V$. The compact closed maps are defined as follows:

Given a vector space $V$ with basis $\{\vec{e}_i\}_{i}$,

$$ \eta_V^l = \eta_V^r : \mathbb{R} \to V \otimes V \qquad \epsilon_V^l = \epsilon_V^r : V \otimes V \to \mathbb{R} \\ 1 \mapsto \sum_i \vec{e}_i \otimes \vec{e}_i \qquad \sum_{ij} c_{ij} \vec{v}_i \otimes \vec{w}_i \mapsto \sum_{ij} c_{ij} \langle \vec{v}_i | \vec{w}_i \rangle $$

**Categorical representation of meaning space.** The tensor in **FVect** is commutative up to isomorphism. This causes the left and the right adjoints to be the same, and thus the left and the right compact closed maps to coincide. Thus **FVect** by itself cannot take the effect of word ordering on meaning into account. [7, 10] propose a way around this obstacle by considering the product category **FVect** × **P** where **P** is a pregroup.

Objects in **FVect** × **P** are of the form $(V, p)$, where $V$ is the vector space for the representation of meaning and $p$ is the pregroup type. There exists a morphism $(f, \leq) : (V, p) \to (W, q)$ if there exists a morphism $f : V \to W$ in **FVect** and $p \leq q$ in **P**.

The compact closed structure of **FVect** and **P** lifts componentwise to the product category **FVect** × **P**:

$$ \eta^l : (\mathbb{R}, 1) \to (V \otimes V, p \cdot p^l) \qquad \eta^r : (\mathbb{R}, 1) \to (V \otimes V, p^r \cdot p) \\ \epsilon^l : (V \otimes V, p^l \cdot p) \to (\mathbb{R}, 1) \qquad \epsilon^r : (V \otimes V, p \cdot p^r) \to (\mathbb{R}, 1) $$

**Definition 3.** An object $(V, p)$ in the product category is called a **meaning space**, where $V$ is the vector space in which the meanings $\vec{v} \in V$ of strings of type $p$ live.

**Definition 4.** *From-meanings-of-words-to-the-meaning-of-the-sentence map.* Let $v_1v_2...v_n$ be a string of words, each $v_i$ with a meaning space representation $\vec{v}_i \in (V_i, p_i)$. Let $x \in P$ be a pregroup type such that $[p_1p_2...p_n \leq x]$ Then the meaning vector for the string is $\vec{v_1v_2...v_n} := f(\vec{v_1} \otimes \vec{v_2} \otimes ... \otimes \vec{v_n}) \in (W, x)$, where $f$ is defined to be the application of the compact closed maps obtained from the reduction $[p_1p_2...p_n \leq x]$ to the composite vector space $V_1 \otimes V_2 \otimes ... \otimes V_n$.

This framework uses the maps of the pregroup reductions and the elements of objects in **FVect**. The diagrammatic calculus provides a tool to reason about both. As an example, take the sentence “John likes Mary”. It has the pregroup type $nn^r sn^l n$, and the vector representations $\overrightarrow{John}$, $\overrightarrow{Mary} \in V$ and $\overrightarrow{likes} \in V \otimes S \otimes V$. The morphism in **FVect** × **P** corresponding to the map defined in **Definition 4** is of type $(V \otimes (V \otimes S \otimes V) \otimes V, nn^r sn^l n) \to (S, s)$. From the pregroup reduction $[nn^r sn^l n \to s]$ we obtain the compact closed maps $\epsilon^r 1 \epsilon^l$. In **FVect** this translates into $\epsilon_V \otimes 1_S \otimes \epsilon_V : V \otimes (V \otimes S \otimes V) \otimes V \to S$. This map, when applied to $\overrightarrow{John} \otimes \overrightarrow{likes} \otimes \overrightarrow{Mary}$, has the following depiction in the diagrammatic calculus:
<figure><img src="image_7.png" /></figure>

Note that this construction treats the verb 'likes' essentially as a relation that takes two inputs of type $V$, and outputs a vector of type $S$. For the explicit calculation, note that $\overrightarrow{likes} = \sum_{ijk} c_{ijk} \overrightarrow{v_i} \otimes \overrightarrow{s_j} \otimes \overrightarrow{v_k}$, where $\{\overrightarrow{v_i}\}_i$ is an orthonormal basis for $V$ and $\{\overrightarrow{s_j}\}_j$ is an orthonormal basis for $S$. Then

$$ \overrightarrow{John \ likes \ Mary} = \epsilon_V \otimes 1_S \otimes \epsilon_V (\overrightarrow{John} \otimes \overrightarrow{likes} \otimes \overrightarrow{Mary}) \quad (1) $$

$$ = \sum_{ijk} \langle \overrightarrow{John} | \overrightarrow{v_i} \rangle \overrightarrow{s_j} \langle \overrightarrow{v_k} | \overrightarrow{Mary} \rangle \quad (2) $$

The reductions in diagrammatic calculus help reduce the final calculation to a simpler term. The non-reduced reduction, when expressed in dirac notation is $(\epsilon_V^r | \otimes 1_S \otimes \langle \epsilon_V^l |) \circ |\overrightarrow{John} \otimes \overrightarrow{likes} \otimes \overrightarrow{Mary}|$. But we can swing $\overrightarrow{John}$ and $\overrightarrow{Mary}$ in accord with the reduction rules in the diagrammatic calculus. The diagram then reduces to:

<figure><img src="image_8.png" /></figure>

This results in a simpler expression that needs to be calculated: $(\overrightarrow{John} | \otimes 1_S \otimes \langle \overrightarrow{Mary} |) \circ |\overrightarrow{likes}|$.

### 3 Density Matrices as Elements of a Compact Closed Category

Recall that in $FVect$, vectors $|\vec{v}\rangle \in V$ are in one-to-one correspondence with morphisms of type $v: I \to V$. Likewise, pure states of the form $|\vec{v}\rangle\langle\vec{v}|$ are in one-to-one correspondence with morphisms $v \circ v^\dagger: V \to V$ such that $v^\dagger \circ v = \text{id}_I$, where $v^\dagger$ denotes the adjoint of $v$ (notice that this corresponds to the condition that $\langle v|v \rangle = 1$). A general (mixed) state $\rho$ is a positive morphism of the form $\rho: V \to V$. One can re-express the mixed states $\rho: V \to V$ as elements $\rho: I \to V^* \otimes V$. Here $V^* = V^l = V^r = V$.

**Definition 5.** *f is a completely positive map if f is positive for any positive operator A, and $(\text{id}_V \otimes f)B$ is positive for any positive operator B and any space V.*

Completely positive maps in $FVect$ form a monoidal category [24]. Thus one can define a new category $CPM(FVect)$ where the objects of $CPM(FVect)$ are the same as those of $FVect$, and morphisms $A \to B$ in $CPM(FVect)$ are completely positive maps $A^* \otimes A \to B^* \otimes B$ in $FVect$. The elements $I \to A$ in $CPM(FVect)$ are of the form $I^* \otimes I \to A^* \otimes A$ in $FVect$, providing a monoidal category with density matrices as its elements.
**CPM(FVect) in graphical calculus.** A morphism $\rho : A \to A$ is positive if and only if there exists a map $\sqrt{\rho}$ such that $\rho = \sqrt{\rho}^\dagger \circ \sqrt{\rho}$. In **FVect**, the isomorphism between $\rho : A \to A$ and $\lceil \rho \rceil : I \to A^* \otimes A$ is provided by $\eta^l = \eta^r$. The graphical representation of $\rho$ in **FVect** then becomes:

<figure><img src="image_9.png" /></figure>

Notice that the categorical definition of a positive morphism coincides with the definition of a positive operator in a vector space, where $\sqrt{\rho}$ is the square root of the operator.

The graphical depiction of completely positive morphisms come from the following theorem:

**Theorem 1.** (Stinespring Dilation Theorem) $f : A^* \otimes A \to B^* \otimes B$ is completely positive if and only if there is an object $C$ and a morphism $\sqrt{f} : A \to C \otimes B$ such that the following equation holds:

<figure><img src="image_10.png" /></figure>

$\sqrt{f}$ and $C$ here are not unique. For the proof of the theorem see [24].

**Theorem 2.** **CPM(FVect)** is a compact closed category where as in **FVect**, $V^r = V^l = V$ and the compact closed maps are defined to be:

$$ \eta^l = (\eta_V^r \otimes \eta_V^l) \circ (\mathbf{1}_V \otimes \sigma \otimes \mathbf{1}_V) \qquad \eta^r = (\eta_V^l \otimes \eta_V^r) \circ (\mathbf{1}_V \otimes \sigma \otimes \mathbf{1}_V) $$

$$ \epsilon^l = (\mathbf{1}_V \otimes \sigma \otimes \mathbf{1}_V) \circ (\epsilon_V^r \otimes \epsilon_V^l) \qquad \epsilon^r = (\mathbf{1}_V \otimes \sigma \otimes \mathbf{1}_V) \circ (\epsilon_V^l \otimes \epsilon_V^r) $$

where $\sigma$ is the swap map defined as $\sigma(v \otimes w) = (w \otimes v)$.

**Proof.** The graphical construction of the compact closed maps boils down to doubling the objects and the wires. The identities are proved by adding bends in the wires. Consider the diagram for $\eta^r$:

<figure><img src="image_11.png" /></figure>
These maps satisfy the axioms of compact closure since the components do.

The concrete compact closed maps are as follows:

$$
\begin{align*}
\eta^l &= \eta^r : \mathbb{R} \to (V \otimes V) \otimes (V \otimes V) \\
\therefore 1 &\mapsto \sum_i \vec{e}_i \otimes \vec{e}_i \otimes \sum_j \vec{e}_j \otimes \vec{e}_j \\
\epsilon^l &= \epsilon^r : (V \otimes V) \otimes (V \otimes V) \to \mathbb{R} \\
\therefore &\sum_{ijkl} c_{ijkl} \langle \vec{v}_i | \vec{w}_j \rangle \otimes \langle \vec{w}_j | \vec{p}_l \rangle \mapsto \sum_{ijkl} c_{ijkl} \langle \vec{v}_i | \vec{u}_k \rangle \otimes \langle \vec{w}_j | \vec{p}_l \rangle
\end{align*}
$$

Let $\rho : V_1 \otimes V_2 \otimes \dots \otimes V_n \to V_1 \otimes V_2 \otimes \dots \otimes V_n$ be a density operator defined on an arbitrary composite space $V_1 \otimes V_2 \otimes \dots \otimes V_n$. Then it has the density matrix representation $\rho : I \to (V_1 \otimes V_2 \otimes \dots \otimes V_n)^* \otimes (V_1 \otimes V_2 \otimes \dots \otimes V_n)$. Since the underlying category $\mathbf{FVect}$ is symmetric, it has the swap map $\sigma$. This provides us with the isomorphism:

$$ (V_1 \otimes V_2 \otimes \dots \otimes V_n)^* \otimes (V_1 \otimes V_2 \otimes \dots \otimes V_n) \sim (V_1^* \otimes V_1) \otimes (V_2^* \otimes V_2) \otimes \dots \otimes (V_n^* \otimes V_n) $$

So $\rho$ can be equivalently expressed as $\rho : I \to (V_1^* \otimes V_1) \otimes (V_2^* \otimes V_2) \otimes \dots \otimes (V_n^* \otimes V_n)$. With this addition, we can simplify the diagrams used to express density matrices by using a single thick wire for the doubled wires. Doubled compact closed maps can likewise be expressed by a single thick wire.

<figure><img src="image_12.png" /></figure>

The diagrammatic expression of a from-meanings-of-words-to-the-meaning-of-the-sentence map using density matrices will therefore look exactly like the depiction of it in $\mathbf{FVect}$, but with thick wires.

## 4 Using Density Matrices to Model Meaning

If one wants to use the full power of density matrices in modelling meaning, one needs to establish an interpretation for the distinction between *mixing* and *superposition* in the context of linguistics. Let *contextual features* be the salient, quantifiable features of the contexts a word is observed in. Let the basis of the space be indexed by such contextual features. Individual *contexts*, such as words in an *n*-word window of a text, can be represented as the superposition of the bases corresponding to the contextual features observed in it. So each context corresponds to a pure state. Words are then probability distributions over the contexts they appear in. The simple co-occurrence model can be cast as a special case of this more general approach, where features and contexts are the same. Then all word meanings are mixtures of basis vectors, and they all commute with each other.
**Similarity for density matrices.** Fidelity is a good measure of similarity between two density matrix representations of meaning because of its properties listed below.

**Definition 6.** The fidelity of two density operators $\rho$ and $\sigma$ is $F(\rho, \sigma) := \text{tr}\sqrt{\rho^{1/2}\sigma\rho^{1/2}}$.

Some useful properties of fidelity are:

1. $F(\rho, \sigma) = F(\sigma, \rho)$

2. $0 \le F(\rho, \sigma) \le 1$

3. $F(\rho, \sigma) = 1$ if and only if $\rho = \sigma$

4. If $|\phi\rangle\langle\phi|$ and $|\psi\rangle\langle\psi|$ are two pure states, their fidelity is equal to $|\langle\phi|\psi\rangle|$.

These properties ensure that if the representations of two words are not equal to each other, they will not be judged perfectly similar, and, if two words are represented as projections onto one dimensional subspaces, their similarity value will be equal to the usual cosine similarity of the vectors.

**Entailment for density matrices.** To develop a theory of entailment using density matrices as the basic representations, we assume the following hypothesis:

**Definition 7 (Distributional Hypothesis for Hyponymy).** The meaning of a word $w$ subsumes the meaning of a word $v$ if and only if it is appropriate to use $w$ in all the contexts $v$ is used.

This is a slightly more general version of the *Distributional Inclusion Hypothesis* (DIH) stated in [17]. The difference lies in the additional power the density matrix formalism provides: the distinction between mixing and superposition. Further, DIH only considers whether or not the target word occurs together with the salient distributional feature at all, and ignores any possible statistically significant correlations of features; here again, the density matrix formalism offers a solution.

Note that [12] show that while there is ample evidence for the distributional inclusion hypothesis, this in itself does not necessarily provide a method to detect hyponymy-hypernymy pairs. One of their suggestions for improvement is to consider more than one word in the features, equivalent to what we do here by taking correlations into account in a co-occurrence space where the bases are context words.

Relative entropy quantifies the distinguishability of one distribution from another. The idea of using relative entropy to model hyponymy is based on the assumption that the distinguishability of one word from another given its usual contexts provides us with a good metric for hyponymy. For example, if one is given a sentence with the word *dog* crossed out, it will be not be possible for sure to know whether the crossed out word is not *animal* just from the context (except perhaps very particular decelerational sentences which rely on world knowledge, such as ‘All – bark’.)

**Definition 8.** The (quantum) relative entropy of two density matrices $\rho$ and $\sigma$ is $N(\rho||\sigma) := \text{tr}(\rho\log\rho) - \text{tr}(\rho\log\sigma)$, where $0 \log 0 = 0$ and $x \log 0 = \infty$ when $x \ne 0$ by convention.

**Definition 9.** The representativeness between $\rho$ and $\sigma$ is $R(\rho, \sigma) := 1/(1 + N(\rho||\sigma))$, where $N(\rho||\sigma)$ is the quantum relative entropy between $\rho$ and $\sigma$.
Quantum relative entropy is always non-negative. For two density matrices $\rho$ and $\sigma$, $N(\rho||\sigma) = \infty$ if $\text{supp}(\rho) \cap \text{ker}(\sigma) \neq 0$, and is finite otherwise. The following is a direct consequence of these properties:

**Corollary 1.** For all density matrices $\rho$ and $\sigma$, $R(\rho, \sigma) \le 1$ with equality if and only if $\rho = \sigma$, and $0 \le R(\rho, \sigma)$ with equality if and only if $\text{supp}(\rho) \cap \text{ker}(\sigma) \neq 0$

The second part of the corollary reflects the idea that if there is a context in which it is appropriate to use $v$ but not $w$, then $v$ is perfectly distinguishable from $w$. Such contexts are exactly those that fall within $\text{supp}(\rho) \cap \text{ker}(\sigma)$.

*Characterizing hyponyms.* The quantitative measure on density matrices given by representativeness provide a qualitative preorder on meaning representations as follows:

$$ \rho \prec \sigma \text{ if } R(\rho, \sigma) > 0 $$

$$ \rho \sim \sigma \text{ if } \prec \sigma \text{ and } \sigma \prec \rho $$

**Proposition 1.** The following are equivalent:

$$ 1. \rho \prec \sigma $$

$$ 2. \text{supp}(\rho) \subseteq \text{supp}(\sigma) $$

$$ 3. \text{There exists a positive operator } \rho' \text{ and } p > 0 \text{ such that } \sigma = p\rho + \rho' $$

*Proof.* (1) $\Rightarrow$ (2) and (2) $\Rightarrow$ (1) follow directly from Corollary 1.

(2) $\Rightarrow$ (3) since $\text{supp}(\rho) \subseteq \text{supp}(\sigma)$ implies that there exists a $p > 0$ such that $\sigma - p\rho$ is positive. Setting $\rho' = \sigma - p\rho$ gives the desired equality.

$$ (3) \Rightarrow (2) \text{ since } p > 0, \text{ and so } \text{supp}(\rho) \subseteq \text{supp}(\sigma) = \text{supp}(p\rho + \rho'). $$

The equivalence relation $\sim$ groups any two density matrices $\rho$ and $\sigma$ with $\text{supp}(\rho) = \text{supp}(\sigma)$ into the same equivalence class, thus maps the set of density matrices on a Hilbert space $\mathcal{H}$ onto the set of projections on $\mathcal{H}$. The projections are in one-to-one correspondence with the subspaces of $\mathcal{H}$ and they form an orthomodular lattice, providing a link to the logical structure of the Hilbert space [26] aims to exploit by using density matrices in IR.

Let $\hat{w}$ and $\hat{v}$ be density matrix representations of the words $v$ and $w$. Then $v$ is a hyponym of $w$ in this model if $\hat{v} \prec \hat{w}$ and $\hat{v} \sim \hat{w}$.

Notice that even though this ordering on density matrices extracts a *yes/no* answer for the question “is $v$ a hyponym of $w$?”, the existence of the quantitative measure lets us to also quantify the extent to which $v$ is a hyponym of $w$. This provides some flexibility in characterizing hyponymy through density matrices in practice. Instead of calling $v$ a hyponym of $w$ even when $R(\hat{v}, \hat{w})$ gets arbitrarily small, one can require the representativeness to be above a certain threshold $\epsilon$. This modification, however, has the down side of causing the transivity of hyponymy to fail.
## 5 From meanings of words to the meanings of sentences passage

As in the case for $FVect \times P$, $CPM(FVect) \times P$ is a compact closed category, where the compact closed maps of $CPM(FVect)$ and $P$ lift component-wise to the product category.

**Definition 10.** A meaning space in this new category is a pair $(V^* \otimes V, p)$ where $V^* \otimes V$ is the space in which density matrices $v: I \to V^* \otimes V$ of the pregroup type $p$ live.

**Definition 11.** Let $v_1v_2\dots v_n$ be a string of words, each $v_i$ with a meaning space representation $\hat{v}_i \in (V_i^* \otimes V_i, p_i)$. Let $x \in P$ be a pregroup type such that $[p_1p_2\dots p_n \le x]$. Then the meaning density matrix for the string is defined as:

$$v_1\hat{v}_2\dots v_n := f(\hat{v}_1 \otimes \hat{v}_2 \otimes \dots \otimes \hat{v}_n) \in (W^* \otimes W, x)$$

where $f$ is defined to be the application of the compact closed maps obtained from the reduction $[p_1p_2\dots p_n \le x]$ to the composite density matrix space $(V_1 \otimes V_1^*) \otimes (V_2^* \otimes V_2) \otimes \dots \otimes (V_n^* \otimes V_n)$.

From a high level perspective, the reduction diagrams for $CPM(FVect) \times P$ look no different than the original diagrams for $FVect \times P$, except that we depict them with thick instead of thin wires. Consider the previous example: “John likes Mary”. It has the pregroup type $n(n^r sn^l)n$, and the compact closed maps obtained from the pregroup reduction is $(\epsilon^r \otimes 1 \otimes \epsilon^l)$.

One can also depict the diagram together with the internal anatomy of the density representations in $FVect$:

<figure><img src="image_13.png" /></figure>

The graphical reductions for compact closed categories can be applied to the diagram, establishing $(\epsilon^r \otimes 1 \otimes \epsilon^l)(\widehat{John} \otimes \widehat{likes} \otimes \widehat{Mary}) = (\widehat{John} \otimes 1 \otimes \widehat{Mary}) \circ \widehat{likes}$.

As formalised in natural logic, one expects that if the subject and object of a sentence are common nouns which are, together with the verb of the sentence, moreover, upward monotone, then if these are replaced by their hyponyms, then the meanings of the original and the modified sentences would preserve this hyponymy. The following proposition shows that the sentence meaning map for simple transitive sentences achieves exactly that.

**Theorem 3.** If $\rho, \sigma, \delta, \gamma \in (N^* \otimes N, n)$, $\alpha, \beta \in (N^* \otimes N \otimes S^* \otimes S \otimes N^* \otimes N, n^l sn^r$ $\rho \prec \sigma, \delta \prec \gamma$ and $\alpha \prec \beta$ then

$$f(\rho \otimes \alpha \otimes \delta) \prec f(\sigma \otimes \beta \otimes \gamma)$$
where $f$ is the from-meanings-of-words-to-the-meaning-of-the-sentence map in definition 11.

*Proof.* If $\rho \prec \sigma$, $\delta \prec \gamma$, and $\alpha \prec \beta$, then there exists a positive operator $\rho'$ and $r > 0$ such that $\sigma = r\rho + \rho'$, a positive operator $\delta'$ and $d > 0$ such that $\gamma = d\delta + \delta'$ and a positive operator $\alpha'$ and $a > 0$ such that $\beta = a\alpha + \alpha'$ by Proposition 1. Then

$$ 
\begin{aligned}
f(\sigma \otimes \beta \otimes \gamma) &= (\epsilon^r \otimes 1 \otimes \epsilon^l)(\sigma \otimes \beta \otimes \gamma) \\
&= (\sigma \otimes 1 \otimes \gamma) \circ \beta \\
&= ((r\rho + \rho') \otimes 1 \otimes (d\delta + \delta')) \circ (a\alpha + \alpha') \\
&= (r\rho \otimes 1 \otimes d\delta) \circ (a\alpha + \alpha') + (\rho' \otimes 1 \otimes \delta') \circ (a\alpha + \alpha') \\
&= (r\rho \otimes 1 \otimes d\delta) \circ a\alpha + (r\rho \otimes 1 \otimes d\delta) \circ \alpha' + (\rho' \otimes 1 \otimes \delta') \circ (a\alpha + \alpha'), \\
f(\rho \otimes \alpha \otimes \delta) &= (\rho \otimes 1 \otimes \delta) \circ \alpha
\end{aligned}
$$

since $r, d, a \neq 0$, $\text{supp}(f(\rho \otimes \alpha \otimes \delta)) \subseteq \text{supp}(f(\sigma \otimes \beta \otimes \gamma))$, which by Proposition 1 proves the theorem.

# 6 Truth Theoretic Examples

We present several examples that demonstrate the application of the from-meanings-of-words-to-the-meaning-of-sentence map, where the initial meaning representations of words are density matrices, and explore how the hierarchy on nouns induced by their density matrix representations carry over to a hierarchy in the sentence space.

## 6.1 Entailment between nouns

Let “lions”, “sloths”. “plants” and “meat” have one dimensional representations in the noun space of our model:

$$ \widehat{lions} = |\overrightarrow{lions}\rangle\langle lions| $$

$$ \widehat{meat} = |\overrightarrow{meat}\rangle\langle meat| $$

$$ \widehat{slaths} = |\overrightarrow{slaths}\rangle\langle slaths| $$

$$ \widehat{plants} = |\overrightarrow{plants}\rangle\langle plants| $$

Let the representation of “mammals” be a mixture of one dimensional representations of individual animals:

$$ \widehat{mammals} = 1/2|\overrightarrow{lions}\rangle\langle lions| + 1/2|\overrightarrow{slaths}\rangle\langle slaths| $$

Notice that

$$ 
\begin{aligned}
N(\widehat{lions}||\widehat{mammals}) &= \text{tr}(\widehat{lions} \log \widehat{lions}) - \text{tr}(\widehat{lions} \log \widehat{mammals}) \\
&= \log 1 - \frac{1}{2} \log \frac{1}{2} = 1
\end{aligned}
$$

Hence $R(\widehat{lions}, \widehat{mammals}) = 1/2$. For the other direction, since the intersection of the support of mammals and the kernel of $\widehat{lions}$ is non-empty, $R(\widehat{mammals}, \widehat{lions}) = 0$. This confirms that $\widehat{lions} \prec \widehat{mammals}$.
## 6.2 Entailment between sentences in one dimensional truth theoretic space

Consider a sentence space that is one dimensional, where 1 stands for true and 0 for false. Let sloths eat plants and lions eat meat; this is represented as follows

$$ \begin{aligned} \widehat{eat} &= (|\overrightarrow{ sloths }| |\overrightarrow{ plants }| + |\overrightarrow{ lions }| |\overrightarrow{ meat }|) (\langle \overrightarrow{ sloths }| \langle \overrightarrow{ plants }| + \langle \overrightarrow{ lions} | \langle \overrightarrow{ meat }|) \\ &\approx (|\overrightarrow{ sloths }| \langle \overrightarrow{ sloths }| \otimes |\overrightarrow{ plants }| \langle \overrightarrow{ plants }|) + (|\overrightarrow{ sloths }| \langle \overrightarrow{ lions }| \otimes |\overrightarrow{ plants }| \langle \overrightarrow{ meat }|) + \\ &\quad (|\overrightarrow{ lions }| \langle \overrightarrow{ sloths }| \otimes |\overrightarrow{ meat }| \langle \overrightarrow{ plants }|) + (|\overrightarrow{ lions }| \langle \overrightarrow{ lions }| \otimes |\overrightarrow{ meat }| \langle \overrightarrow{ meat }|) \end{aligned} $$

The above is the density matrix representation of a pure composite state that relate “sloths” to “plants” and “lions” to “meat”. If we fix the bases {$\overrightarrow{lions}, \overrightarrow{sloths}$} for $N_1$, and {$\overrightarrow{meat}, \overrightarrow{plants}$} for $N_2$, we will have $\widehat{eat} : N_1 \otimes N_1 \rightarrow N_2 \otimes N_2$ with the following matrix representation:

$$ \begin{pmatrix} 1 & 0 & 0 & 1 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 1 & 0 & 0 & 1 \end{pmatrix} $$

“Lions eat meat”. This is a transitive sentence, so as before, it has the pregroup type: $nn^l sn^r n$. Explicit calculations for its meaning give:

$$ \begin{aligned} & (\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{lions} \otimes \widehat{eat} \otimes \widehat{meat}) \\ &= \langle \overrightarrow{lions} | \overrightarrow{sloths} \rangle^2 \langle \overrightarrow{plants} | \overrightarrow{meat} \rangle^2 + \\ & \quad \langle \overrightarrow{lions} | \overrightarrow{sloths} \rangle \langle \overrightarrow{lions} | \overrightarrow{lions} \rangle \langle \overrightarrow{meat} | \overrightarrow{meat} \rangle \langle \overrightarrow{plants} | \overrightarrow{meat} \rangle + \\ & \quad \langle \overrightarrow{lions} | \overrightarrow{lions} \rangle \langle \overrightarrow{lions} | \overrightarrow{sloths} \rangle \langle \overrightarrow{meat} | \overrightarrow{meat} \rangle \langle \overrightarrow{plants} | \overrightarrow{meat} \rangle + \\ & \quad \langle \overrightarrow{lions} | \overrightarrow{lions} \rangle^2 \langle \overrightarrow{meat} | \overrightarrow{meat} \rangle^2 \\ &= 1 \end{aligned} $$

“Sloths eat meat”. This sentence has a very similar calculation to the one above with the resulting meaning:

$$ (\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{sloths} \otimes \widehat{eat} \otimes \widehat{meat}) = 0 $$

“Mammals eat meat”. This sentence has the following meaning calculation:

$$ \begin{aligned} & (\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{mammals} \otimes \widehat{eat} \otimes \widehat{meat}) = \\ & (\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)((\frac{1}{2}\widehat{lions} + \frac{1}{2}\widehat{sloths}) \otimes \widehat{eat} \otimes \widehat{meat}) = \\ & \frac{1}{2}(\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{lions} \otimes \widehat{eat} \otimes \widehat{meat}) + \frac{1}{2}(\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{sloths} \otimes \widehat{eat} \otimes \widehat{meat}) = \frac{1}{2} \end{aligned} $$

The resulting meaning of this sentence is a mixture of “lions eat meat”, which is true, and “sloths eat meat” which is false. Thus the value 1/2 can be interpreted as being neither completely true or completely false: the sentence “mammals eat meat” is true for certain mammals and false for others.
## 6.3 Entailment between sentences in two dimensional truth theoretic space

The two dimensional truth theoretic space is set as follows:

$$ true \equiv |0\rangle \equiv \begin{pmatrix} 1 \\ 0 \end{pmatrix} \qquad false \equiv |1\rangle \equiv \begin{pmatrix} 0 \\ 1 \end{pmatrix} $$

The corresponding true and false density matrices are $|0\rangle\langle 0|$ and $|1\rangle\langle 1|$.

In the two dimensional space, the representation of “eats” is set as follows. Let $A = \{\text{lions}, \text{slaths}\}$ and $B = \{\text{meat}, \text{plants}\}$, then

$$ \widehat{eat} \equiv \sum_{\substack{a_1, a_2 \in A \\ b_1, b_2 \in B}} |\vec{a}_1\rangle\langle\vec{a}_2| \otimes |\vec{x}\rangle\langle\vec{x}| \otimes |\vec{b}_1\rangle\langle\vec{b}_2| $$

where

$$ |x\rangle \equiv \begin{cases} |0\rangle & \text{if } |a_1\rangle|b_1\rangle, |a_2\rangle|b_2\rangle \in \{\overrightarrow{\text{lions}}|\overrightarrow{\text{meat}}\rangle, \overrightarrow{\text{slaths}}|\overrightarrow{\text{plants}}\rangle\} \\ |1\rangle & \text{otherwise} \end{cases} $$

The generalized matrix representation of this verb in the spirit of [14] is:

$$ \begin{pmatrix} 1 & 0 & 0 & 1 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 1 & 1 & 1 \\ 0 & 0 & 0 & 0 & 1 & 1 & 1 \\ 1 & 0 & 0 & 1 & 0 & 1 & 0 \end{pmatrix} $$

“Lions eat meat”. The calculation for the meaning of this sentence is almost exactly the same as the case of the one dimensional meaning, only the result is not the scalar that stands for true but its density matrix:

$$ (\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{\text{lions}} \otimes \widehat{\text{eat}} \otimes \widehat{\text{meat}}) = |0\rangle\langle 0| $$

“Sloths eat meat”. Likewise, the calculation for the meaning of this sentence returns false:

$$ (\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{\text{slaths}} \otimes \widehat{\text{eat}} \otimes \widehat{\text{meat}}) = |1\rangle\langle 1| $$

“Mammals eat meat”. As we saw before, this sentence has the meaning that is the mixture of “Lions eat meat” and “Sloths eat meat”; here, this is expressed as follows:

$$ \begin{align*} & (\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{\text{mammals}} \otimes \widehat{\text{eat}} \otimes \widehat{\text{meat}}) \\ &= \frac{1}{2}(\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{\text{lions}} \otimes \widehat{\text{eat}} \otimes \widehat{\text{meat}}) + \frac{1}{2}(\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{\text{slaths}} \otimes \widehat{\text{eat}} \otimes \widehat{\text{meat}}) \\ &= \frac{1}{2}|1\rangle\langle 1| + \frac{1}{2}|0\rangle\langle 0| \end{align*} $$

So in a two dimensional truth theoretic model, “Mammals eat meat” give the completely mixed state in the sentence space, which has maximal entropy. This is equivalent to
saying that we have no real knowledge whether mammals in general eat meat or not. Even if we are completely certain about whether individual mammals that span our space for “mammals” eat meat, this information differs uniformly within the members of the class, so we cannot generalize.

Already with a two dimensional truth theoretic model, the relation $\widehat{\text{lions}} \prec \widehat{\text{mammals}}$ carries over to sentences. To see this, first note that we have

$$ 
\begin{aligned}
N(\widehat{\text{lions eat meat}} | \widehat{\text{mammals eat meat}}) &= N\left(|0\rangle\langle0| \middle| \frac{1}{2}|0\rangle\langle0| + \frac{1}{2}|1\rangle\langle1|\right) \\
&= (|0\rangle\langle0|) \log(|0\rangle\langle0|) - (|0\rangle\langle0|) \log\left(\frac{1}{2}|0\rangle\langle0| + \frac{1}{2}|1\rangle\langle1|\right) = 1
\end{aligned}
$$

In the other direction, we have $N(\widehat{\text{mammals eat meat}} | \widehat{\text{lions eat meat}}) = \infty$, since the intersection of the support of the first argument and the kernel of the second argument is non-trivial. These lead to the following representativeness results between sentences:

$$ R(\widehat{\text{lions eat meat}}, \widehat{\text{mammals eat meat}}) = 1/2 $$

$$ R(\widehat{\text{mammals eat meat}} | \widehat{\text{lions eat meat}}) = 0 $$

As a result we obtain:

$$ \widehat{\text{lions eat meat}} \prec \widehat{\text{mammals eat meat}} $$

Since these two sentences share the same verb phrase, from-meaning-of-words-to-the-meaning-of-sentence map carries the hyponymy relation in the subject words of the respective sentences to the resulting sentence meanings. By using the density matrix representations of word meanings together with the categorical map from the meanings of words to the meanings of sentences, the knowledge that a lion is an animal lets us infer that “mammals eat meat” implies “lions eat meat”:

$$ (\widehat{\text{lions}} \prec \widehat{\text{mammals}}) \rightarrow (\widehat{\text{lions eat meat}} \prec \widehat{\text{mammals eat meat}}) $$

“Dogs eat meat”. To see how the completely mixed state differs from a perfectly correlated but pure state in the context of linguistic meaning, consider a new noun $\overrightarrow{\text{dog}} = |\overrightarrow{\text{dog}}\rangle\langle\overrightarrow{\text{dog}}|$ and redefine eat in terms of the bases $\{\overrightarrow{\text{lions}}, \overrightarrow{\text{dogs}}\}$ and $\{\overrightarrow{\text{meat}}, \overrightarrow{\text{plants}}\}$, so that it will reflect the fact that dogs eat both meat and plants. We define “eat” so that it results in the value of being “half-true half-false” when it takes “dogs” as subject and “meat” or “plants” as object. The value “half-true half-false” is the superposition of true and false: $\frac{1}{2}|0\rangle + \frac{1}{2}|1\rangle$. With this assumptions, $\widehat{\text{eat}}$ will still be a pure state with the following representation in FVect:

$$ 
\begin{aligned}
|\widehat{\text{eat}}\rangle &= |\overrightarrow{\text{lions}}\rangle \otimes |0\rangle \otimes |\overrightarrow{\text{meat}}\rangle + |\overrightarrow{\text{lions}}\rangle \otimes |1\rangle \otimes |\overrightarrow{\text{plants}}\rangle + \\
& \qquad |\overrightarrow{\text{dogs}}\rangle \otimes (\frac{1}{2}|0\rangle + \frac{1}{2}|1\rangle) \otimes |\overrightarrow{\text{meat}}\rangle + |\overrightarrow{\text{dogs}}\rangle \otimes (\frac{1}{2}|0\rangle + \frac{1}{2}|1\rangle) \otimes |\overrightarrow{\text{plants}}\rangle
\end{aligned}
$$
Hence, the density matrix representation of “eat” becomes:

$$ \widehat{eat} = |\overrightarrow{eat}\rangle\langle\overrightarrow{eat}| $$

The calculation for the meaning of the sentence is as follows:

$$ \begin{align*} & (\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{dogs} \otimes \widehat{eat} \otimes \widehat{meat}) \\ &= (\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(|\overrightarrow{dogs}\rangle\langle\overrightarrow{dogs}| \otimes |\overrightarrow{eat}\rangle\langle\overrightarrow{eat}| \otimes |\overrightarrow{meat}\rangle\langle\overrightarrow{meat}|) \\ &= (\frac{1}{2}|0\rangle + \frac{1}{2}|1\rangle)(\frac{1}{2}\langle0| + \frac{1}{2}\langle1|) \end{align*} $$

So in this case, we are certain that it is half-true and half-false that dogs eat meat. This is in contrast with the completely mixed state we got from “Mammals eat meat”, for which the truth or falsity of the sentence was entirely unknown.

“Mammals eat meat”, again. Let “mammals” now be defined as:

$$ \widehat{mammals} = \frac{1}{2}\widehat{lions} + \frac{1}{2}\widehat{dogs} $$

The calculation for the meaning of this sentence gives:

$$ \begin{align*} & (\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{mammals} \otimes \widehat{eat} \otimes \widehat{meat}) \\ &= \frac{1}{2}(\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{lions} \otimes \widehat{eat} \otimes \widehat{meat}) + \frac{1}{2}(\epsilon_N^l \otimes 1_S \otimes \epsilon_N^r)(\widehat{dogs} \otimes \widehat{eat} \otimes \widehat{meat}) \\ &= \frac{3}{4}|0\rangle\langle0| + \frac{1}{4}|0\rangle\langle1| + \frac{1}{4}|1\rangle\langle0| + \frac{1}{4}|1\rangle\langle1| \end{align*} $$

This time the resulting sentence representation is not completely mixed. This means that we can generalize the knowledge we have from the specific instances of mammals to the entire class to some extent, but still we cannot generalize completely. This is a mixed state, which indicates that even if the sentence is closer to *true* than to *false*, the degree of truth isn't homogeneous throughout the elements of the class. The non-zero non-diagonals indicate that it is also partially correlated, which means that there are some instances of “mammals” for which this sentence is true to a degree, but not completely. The relative similarity measures between *true* and *false* and the sentence can be calculated explicitly using fidelity:

$$ F(|1\rangle\langle1|, \widehat{mammals \ eat \ meat}) = \langle1|\widehat{mammals \ eat \ meat}|1\rangle = \frac{1}{4} $$

$$ F(|0\rangle\langle0|, \widehat{mammals \ eat \ meat}) = \langle0|\widehat{mammals \ eat \ meat}|0\rangle = \frac{3}{4} $$

Notice that these values are different from the values for the representativeness for truth and falsity of the sentence, even thought they are proportional: the more
representative their density matrices, the more similar the sentences are to each other. For example, we have:

$$ N(|1\rangle\langle1| \parallel \text{mammals eat meat}) = \\ \text{tr}(|1\rangle\langle1|) \log(|1\rangle\langle1|) - \text{tr}(|1\rangle\langle1| \log(\frac{3}{4}|0\rangle\langle0| + \frac{1}{4}|0\rangle\langle1| + \frac{1}{4}|1\rangle\langle0| + \frac{1}{4}|1\rangle\langle1|)) \approx 2 $$

Hence, $R(|1\rangle\langle1| \parallel \text{mammals eat meat}) \approx .33$. On the other hand:

$$ N(|0\rangle\langle0| \parallel \text{mammals eat meat}) = \\ \text{tr}(|0\rangle\langle0|) \log(|0\rangle\langle0|) - \text{tr}(|0\rangle\langle0| \log(\frac{3}{4}|0\rangle\langle0| + \frac{1}{4}|0\rangle\langle1| + \frac{1}{4}|1\rangle\langle0| + \frac{1}{4}|1\rangle\langle1|)) \approx 0.41 $$

Hence, $R(|0\rangle\langle0| \parallel \text{mammals eat meat}) \approx 0.71$

# 7 Distributional Examples

The goal of this section is to show how one can obtain density matrices for words using lexical taxonomies and co-occurrence frequencies counted from corpora of text. We show how these density matrices are used in example sentences and how the density matrices of their meanings look like. We compute the representativeness formula for these sentences to provide a proof of concept that this measure does makes sense for data harvested from corpora distributionally and that its application is not restricted to truth-theoretic models. Implementing these constructions on real data and validating them on large scale datasets constitute work in progress.

## 7.1 Entailment between nouns

Suppose we have a noun space $N$. Let the subspace relevant for this part of the example be spanned by lemmas *pub*, *pitcher*, *tonic*. Assume that the (non-normalized version of the) vectors of the atomic words *lager* and *ale* in this subspace are as follows:

$$ \begin{aligned} \overrightarrow{lager} &= 6 \times \overrightarrow{pub} + 5 \times \overrightarrow{pitcher} + 0 \times \overrightarrow{tonic} \\ \overrightarrow{ale} &= 7 \times \overrightarrow{pub} + 3 \times \overrightarrow{pitcher} + 0 \times \overrightarrow{tonic} \end{aligned} $$

Suppose further that we are given taxonomies such as 'beer = lager + ale', harvested from a resource such as WordNet. Atomic words (i.e. leafs of the taxonomy), correspond to *pure* states and their density matrices are the projections onto the one dimensional subspace spanned by $|\overrightarrow{w}\rangle\langle\overrightarrow{w}|$. Non-atomic words (such as *beer*) are also density matrices, harvested from the corpus using a feature-based method similar to that of [12]. This is done by counting (and normalising) the frequency of times a word has co-occurred with a subset $B$ of bases in a window in which other bases (the ones not in $B$) have not occurred.

Formally, for a subset of bases {$b1, b2, ..., bn$}, we collect co-ordinates $C_{ij}$ for each tuple $|bi\rangle|bj\rangle$ and build the density matrix $\sum_{ij} C_{ij} |bi\rangle|bj\rangle$.
For example, suppose we see *beer* six times with just *pub*, seven times with both *pub* and *pitcher*, and none-whatsoever with *tonic*. Its corresponding density matrix will be as follows:

$$ \begin{aligned} \widehat{\text{beer}} &= 6 \times |\overrightarrow{\text{pub}}\rangle \langle \overrightarrow{\text{pub}}| + 7 \times (|\overrightarrow{\text{pub}}\rangle + |\overrightarrow{\text{pitcher}}\rangle)(\langle \overrightarrow{\text{pub}}| + \langle \overrightarrow{\text{pitcher}}|) \\ &= 13 \times |\overrightarrow{\text{pub}}\rangle \langle \overrightarrow{\text{pub}}| + 7 \times |\overrightarrow{\text{pub}}\rangle \langle \overrightarrow{\text{pitcher}}| + 7 \times |\overrightarrow{\text{pitcher}}\rangle \langle \overrightarrow{\text{pub}}| + 7 \times |\overrightarrow{\text{pitcher}}\rangle \langle \overrightarrow{\text{pitcher}}| \end{aligned} $$

To calculate the similarity and representativeness of the word pairs, we first normalize them via the operation $\frac{\rho}{\text{Tr}\rho}$, then apply the corresponding formulae. For example, the degree of similarity between 'beer' and 'lager' using fidelity is as follows:

$$ \text{Tr}\sqrt{\widehat{lager}^{\frac{1}{2}} \cdot \widehat{beer} \cdot \widehat{lager}^{\frac{1}{2}}} = 0.93 $$

The degree of entailment *lager* ≺ *beer* is 0.82 as computed as follows:

$$ \frac{1}{1 + \text{Tr}(\widehat{lager} \cdot \log(\widehat{lager}) - \widehat{lager} \cdot \log(\widehat{beer}))} = 0.82 $$

The degree of entailment *beer* ≺ *lager* is 0, like one would expect.

## 7.2 Entailment between sentences

To see how the entailment between sentences follows from the entailment between words, consider example sentences 'Psychiatrist is drinking lager' and 'Doctor is drinking beer'. For the sake of brevity, we assume the meanings of *psychiatrist* and *doctor* are mixtures of basis elements, as follows:

$$ \begin{aligned} \widehat{\text{psychiatrist}} &= 2 \times |\overrightarrow{\text{patient}}\rangle \langle \overrightarrow{\text{patient}}| + 5 \times |\overrightarrow{\text{mental}}\rangle \langle \overrightarrow{\text{mental}}| \\ \widehat{\text{doctor}} &= 5 \times |\overrightarrow{\text{patient}}\rangle \langle \overrightarrow{\text{patient}}| + 2 \times |\overrightarrow{\text{mental}}\rangle \langle \overrightarrow{\text{mental}}| + 3 \times |\overrightarrow{\text{surger}}\rangle \langle \overrightarrow{\text{surger}}| \end{aligned} $$

The similarity between *psychiatrist* and *doctor* is:

$$ S(\widehat{\text{psychiatrist}}, \widehat{\text{doctor}}) = S(\widehat{\text{doctor}}, \widehat{\text{psychiatrist}}) = 0.76 $$

The representativeness between them is:

$$ R(\widehat{\text{psychiatrist}}, \widehat{\text{doctor}}) = 0.49 \qquad R(\widehat{\text{doctor}}, \widehat{\text{psychiatrist}}) = 0 $$

We build matrices for the verb *drink* following the method of [15]. Intuitively this is as follows: the value in entry (i, j) of this matrix will reflect how typical it is for the verb to have a subject related to the i'th basis and an object related to the j'th basis. We assume that the small part of the matrix that interests us for this example is as follows:

<table><thead><tr><th>drink</th><th>pub</th><th>pitcher</th><th>tonic</th></tr></thead><tbody><tr><th>patient</th><td>4</td><td>5</td><td>3</td></tr><tr><th>mental</th><td>6</td><td>3</td><td>2</td></tr><tr><th>surgery</th><td>1</td><td>2</td><td>1</td></tr></tbody></table>
This representation can be seen as a pure state living in a second order tensor. Therefore the density matrix representation of the same object is $\widehat{drink} = |\widehat{drink}\rangle\langle\widehat{drink}|$, a fourth order tensor. Lifting the simplifications introduced in [15] from vectors to density matrices, we obtain the following linear algebraic closed forms for the meaning of the sentences:

$$ \begin{aligned} \widehat{\text{Psychiatrist is drinking lager}} &= \widehat{\text{drink}} \odot (\widehat{\text{psychiatrist}} \otimes \widehat{\text{lager}}) \\ \widehat{\text{Doctor is drinking beer}} &= \widehat{\text{drink}} \odot (\widehat{\text{doctor}} \otimes \widehat{\text{beer}}) \end{aligned} $$

Applying the fidelity and representativeness formulae to sentence representations, we obtain the following values:

$$ \begin{aligned} S(\widehat{\text{Psychiatrist is drinking lager}}, \widehat{\text{Doctor is drinking beer}}) &= 0.81 \\ R(\widehat{\text{Psychiatrist is drinking lager}}, \widehat{\text{Doctor is drinking beer}}) &= 0.53 \\ R(\widehat{\text{Doctor is drinking beer}}, \widehat{\text{Psychiatrist is drinking lager}}) &= 0 \end{aligned} $$

From the relations $\widehat{\text{psychiatrist}} \prec \widehat{\text{doctor}}$ and $\widehat{\text{lager}} \prec \widehat{\text{beer}}$ we obtain the desired entailment between sentences:

$$ \widehat{\text{Psychiatrist is drinking lager}} \prec \widehat{\text{Doctor is drinking beer}}. $$

The entailment between these two sentences follows from the entailment between their subjects and the entailment between their objects. In the examples that we have considered so far, the verbs of sentences are the same. This is not a necessity. One can have entailment between sentences that do not have the same verbs, but where the verbs entail each other, examples can be found in [2]. The reason we do not present such cases here is lack of space.

# 8 Conclusion and Future Work

The often stated long term goal of compositional distributional models is to merge distributional and formal semantics. However, what formal and distributional semantics *do* with the resulting meaning representations is quite different. Distributional semanticists care about *similarity* while formal semanticists aim to capture *truth* and *inference*. In this work we presented a theory of meaning using basic objects that will not confine us to the realm of only distributional or only formal semantics. The immediate next step is to develop methods for obtaining density matrix representations of words from corpus, that are more robust to statistical noise, and testing the usefulness of the theory in large scale experiments.

The problem of integrating function words such as ‘and’, ‘or’, ‘not’, ‘every’ into a distributional setting has been notoriously hard. We hope that the characterization of compositional distributional entailment on these very simple types of sentences will provide a foundation on which we can define representations of these function words, and develop a more logical theory of compositional distributional meaning.
Bibliography

[1] S. Abramsky and B. Coecke. A categorical semantics of quantum protocols. In *Proceedings of the 19th Annual IEEE Symposium on Logic in Computer Science*, pages 415–425. IEEE Computer Science Press, 2004. arXiv:quant-ph/0402130.

[2] Esma Balkır. Using density matrices in a compositional distributional model of meaning. Master's thesis, University of Oxford, 2014.

[3] Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. Entailment above the word level in distributional semantics. In *Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics*, pages 23–32. Association for Computational Linguistics, 2012.

[4] Islam Beltagy, Cuong Chau, Gemma Boleda, Dan Garrette, Katrin Erk, and Raymond Mooney. Montague meets markov: Deep semantics with probabilistic logical form. In *Second Joint Conference on Lexical and Computational Semantics*, volume 1, pages 11–21. Assocition of Computational Linguists, 2013.

[5] William Blacoe, Elham Kashefi, and Mirella Lapata. A quantum-theoretic approach to distributional semantics. In *Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 847–857, 2013.

[6] Peter D Bruza and Richard Cole. Quantum logic of semantic space: An exploratory investigation of context effects in practical reasoning. *We Will Show Them: Essays in Honour of Dov Gabbay*, pages 339–361, 2005.

[7] Stephen Clark, Bob Coecke, and Mehrnoosh Sadrzadeh. A compositional distributional model of meaning. *Proceedings of the Second Symposium on Quantum Interaction (QI-2008)*, pages 133–140, 2008.

[8] Daoud Clarke. Context-theoretic semantics for natural language: an overview. In *Proceedings of the workshop on geometrical models of natural language semantics*, pages 112–119. Association for Computational Linguistics, 2009.

[9] Bob Coecke. Quantum picturalism. *Contemporary physics*, 51(1):59–83, 2010.

[10] Bob Coecke, Mehrnoosh Sadrzadeh, and Stephen Clark. Mathematical foundations for a compositional distributional model of meaning. *Linguistic Analysis*, 36, 2010.

[11] John R. Firth. A Synopsis of Linguistic Theory, 1930-1955. *Studies in Linguistic Analysis*, pages 1–32, 1957.

[12] Maayan Geffet and Ido Dagan. The distributional inclusion hypotheses and lexical entailment. In *Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics*, pages 107–114. Association for Computational Linguistics, 2005.

[13] Oren Glickman, Ido Dagan, and Moshe Koppel. Web based probabilistic textual entailment. In *Proceedingsof the PASCAL Challenges Workshop on Recognizing Textual Entailment*, 2005.

[14] Edward Grefenstette. *Category-Theoretic Quantitative Compositional Distributional Models of Natural Language Semantics*. PhD thesis, University of Oxford, 2013.
[15] Edward Grefenstette and Mehrnoosh Sadrzadeh. Experimental support for a categorical compositional distributional model of meaning. In *Proceedings of the Conference on Empirical Methods in Natural Language Processing*, pages 1394–1404. Association for Computational Linguistics, 2011.

[16] Aurélie Herbelot and Mohan Ganesalingam. Measuring semantic content in distributional vectors. In *Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics*, volume 2, pages 440–445. Association for Computational Linguistics, 2013.

[17] Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geffet. Directional distributional similarity for lexical inference. *Natural Language Engineering*, 16(04):359–389, 2010.

[18] Joachim Lambek. Type grammars as pregroups. *Grammars*, 4(1):21–39, 2001.

[19] Alessandro Lenci and Giulia Benotto. Identifying hypernyms in distributional semantic spaces. In *Proceedings of the First Joint Conference on Lexical and Computational Semantics*, volume 2, pages 75–79. Association for Computational Linguistics, 2012.

[20] Dekang Lin. An information-theoretic definition of similarity. In *Proceedings of the International Conference on Machine Learning*, pages 296–304, 1998.

[21] Bill MacCartney and Christopher D. Manning. Natural logic for textual inference. In *ACL Workshop on Textual Entailment and Paraphrasing*. Association for Computational Linguistics, 2007.

[22] Robin Piedeleu, Dimitri Kartsaklis, Bob Coecke, and Mehrnoosh Sadrzadeh. Open system categorical quantum semantics in natural language processing. *arXiv preprint arXiv:1502.00831*, 2015.

[23] Enrico Santus, Alessandro Lenci, Qin Lu, and Sabine Schulte Im Walde. Chasing hypernyms in vector spaces with entropy. In *Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics*, volume 2, pages 38–42, 2014.

[24] Peter Selinger. Dagger compact closed categories and completely positive maps. *Electronic Notes in Theoretical Computer Science*, 170:139–163, 2007.

[25] Alessandro Sordoni, Jian-Yun Nie, and Yoshua Bengio. Modeling term dependencies with quantum language models for ir. In *Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval*, pages 653–662. Association for Computational Linguistics, 2013.

[26] Cornelis Joost Van Rijsbergen. *The geometry of information retrieval*. Cambridge University Press, 2004.

[27] Julie Weeds, David Weir, and Diana McCarthy. Characterising measures of lexical distributional similarity. In *Proceedings of the 20th international conference on Computational Linguistics*, number 1015. Association for Computational Linguistics, 2004.