SlowGuess/ACL_2020_OCR huggingface dataset:
ponym of  $Q$  if the region for  $P$  is contained in the region for  $Q$ . Bouraoui et al. (2017) and Vilnis et al. (2018) use this idea for knowledge base completion, and Bouraoui et al. (2020) build on this, using corpus data to identify "conceptual neighbours". In the type-driven tensorial framework (see §2), Bankova et al. (2019) and Lewis (2019) model words as normalised positive operators, with hyponymy defined in terms of subspaces (eigenspaces).
Probability distributions also allow us to define hyponymy, but it is harder than for regions, since a distribution over a smaller region has higher probability density. Vilnis and McCallum (2015) propose using KL-divergence. Athiwaratkun and Wilson (2018) propose a thresholded KL-divergence. In the type-driven tensorial framework, Balkir (2014) proposes using a quantum version of KLdivergence, which can be extended to phrases (Balkir et al., 2015; Sadrzadeh et al., 2018).
However, detecting hyponymy from corpus data remains challenging. Even in recent shared tasks (Bordea et al., 2016; Camacho-Collados et al., 2018), many systems use pattern matching, following Hearst (1992). For example, a string of the form  $X$  such as  $Y$  suggests that  $Y$  is a hyponym of  $X$ . In the above shared tasks, the best performing systems did not rely solely on distributional vectors, but used pattern matching as well.
Although much work remains to be done in developing learning algorithms which can detect hyponymy, I believe that a region-based approach is the most promising. Not only does it give a simple definition, but it is also motivated for other reasons, discussed elsewhere in this paper.
5 Sentence Meaning
In the previous section, I discussed meaning at the level of words. I now turn to challenges in representing meaning at the level of sentences.
5.1 Compositionality
Language is productive – a fluent speaker can understand a completely new sentence, as long as they know each word and each syntactic construction in the sentence. One goal for a semantic model is to be able to derive the meaning of a sentence from its parts, so it can generalise to new combinations. This is known as compositionality.[15]
For vector space models, the challenge is how to compose word vectors to construct phrase representations. If we represent both words and phrases in the same vector space, the challenge is to find a composition function that maps a pair of vectors to a new vector. In the general case, this must be sensitive to word order, since changing word order can change meaning. Mitchell and Lapata (2008, 2010) compare a variety of such functions, but find that componentwise multiplication performs best, despite being commutative, and hence insensitive to word order. The effectiveness of componentwise multiplication and addition has been replicated many times (for example: Baroni and Zamparelli, 2010; Blacoe and Lapata, 2012; Rimell et al., 2016; Czarnowska et al., 2019). However, it is unclear how to adapt it to take word order into account, and Polajnar et al. (2014) show that performance degrades with sentence length.
Alternatively, we can use a sentence space distinct from the word space. This is often done with a task-based perspective – words are combined into sentence representations, which are useful for solving some task. For example, the final state of an RNN can be seen as a representation of the whole sequence. To make the composition more linguistically informed, the network can be defined to follow a tree structure, rather than linear order (for example: Socher et al., 2010, 2012; Tai et al., 2015), or even to learn latent tree structure (for example: Dyer et al., 2016; Maillard and Clark, 2018). Alternatively, a sequence of token representations can be combined using attention, which calculates a weighted sum, as in a Transformer architecture (Vaswani et al., 2017).
Regardless of architecture, the model can be optimised either for a supervised task, such as machine translation (for example: Cho et al., 2014), or for an unsupervised objective, as in an autoencoder (for example: Hermann and Blunsom, 2013) or language model (for example: Peters et al., 2018; Devlin et al., 2019). If we take a task-based perspective, it is difficult to know if the representations will transfer to other tasks. In fact, Changpinyo et al. (2018) find that for some combinations of tasks, training on one task can be harmful for another.
As an alternative to task-based approaches, the tensorial framework mentioned in §2 also uses sentence vectors, but using tensor contraction to
15 Kartsaklis et al. (2013) discuss how composition is often conflated with disambiguation, since composing ambiguous expressions often disambiguates them. Disambiguation can be seen as a kind of contextualisation or context dependence,
which I discuss in §5.3. The focus in this section is on deriving semantic representations for larger expressions.
7441
compose representations based on argument structure. $^{16}$  Polajnar et al. (2015) explore sentence spaces with dimensions defined by co-occurrences.
However, a weakness with the above approaches is that they map sentences to a finite-dimensional space. As we increase sentence length, the number of sentences with distinct meanings increases exponentially. For example, consider relative clauses: the dog chased the cat; the dog chased the cat which caught the mouse; and so on. To keep these meanings distinct, we have two options. If the meanings must be a certain distance apart, the magnitudes of sentence vectors need to increase exponentially with sentence length, so there is enough space to distinguish them.[17] Alternatively, if the meanings can be arbitrarily close, we need to record each dimension to a high precision in order to distinguish the meanings. The fine-grained structure of the space then becomes important, but small changes to model parameters (such as updates during training) would cause drastic changes to this structure. I do not know any work exploring either option. Otherwise, we are forced to view sentence vectors as lossy compression.[18] As Mooney (2014) put it: "You can't cram the meaning of a whole  $\%\&!$ # sentence into a single  $\$ \&!$ #* vector!"
Although compression can be useful for many tasks, full and detailed semantic representations also have their place. This is particularly important at a discourse level: it would be absurd to represent, as vectors of the same dimensionality, both a five-word sentence and the whole English Wikipedia. However, this leaves open the question of how we should represent sentence meaning. In the following section, I turn to logic as a guide.
5.2 Logic
Sentences can express complex thoughts, and build chains of reasoning. Logic formalises this, and one goal for a semantic model is to support the logical notions of truth (discussed in §3.2), and entailment (one proposition following from another).
Vectors do not have logical structure, but can still
be used to provide features for a logical system, for example if entailment is framed as classification: given a premise and hypothesis, the task is to decide if the premise entails the hypothesis, contradicts it, or neither. Datasets include SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018).
However, it is difficult to analyse approaches that do not use an explicit logic. In fact, Gururan-gan et al. (2018) suggest that high performance may be due to annotation artifacts: only using the hypothesis, they achieve  $67\%$  on SNLI and  $53\%$  on MultiNLI, much higher than the majority class baseline ( $34\%$  and  $35\%$ , respectively). Performance on such datasets may therefore overestimate the ability of neural models to perform inference.
To explicitly represent logical structure, there are a few options. One is to build a hybrid system, combining a vector space with a logic. For example, Herbelot and Vecchi (2015) aim to give logical interpretations to vectors. They consider a number of properties (such as: is_edible, has_a_handle, made_of_wood), and for each, they learn a mapping from vectors to values in [0, 1], where 0 means the property applies to no referents, and 1 means it applies to all referents. This is an interesting way to probe what information is available in distributional vectors, but it is unclear how it could be generalised to deal with individual referents (rather than summarising them all), or to deal with complex propositions (rather than single properties).
Garrette et al. (2011) and Beltagy et al. (2016) incorporate a vector space model into a Markov Logic Network (Richardson and Domingos, 2006), a kind of probability logic. If two predicates have high distributional similarity, they add a probabilistic inference rule saying that, if one predicate is true of an entity, the other predicate is likely to also be true. This allows us to use distributional vectors in a well-defined logical model, but it assumes we can interpret similarity in terms of inference (for discussion, see: Erk, 2016). As argued in §3 above, pre-trained vectors may have already lost information, and in the long term, it would be preferable to learn logical representations directly.
Lewis and Steedman (2013) use a classical logic, and cluster predicates that are observed to hold of the same pairs of named entities – for example, write(Rowling, Harry Potter) and author(Rowling, Harry Potter). This uses corpus data directly, rather than pre-trained vectors. However, it would need to be generalised to learn from arbitrary sentences,
16Zanzotto et al. (2015) show how sentence similarity in this framework decomposes in terms of similarity of corresponding parts, because composition and dot products are linear.
This can be formalised information-theoretically. Consider sending a message as a  $D$ -dimensional vector, through a noisy channel. If there is an upper bound  $K$  to the vector's magnitude, the channel has a finite channel capacity. The capacity scales as  $K^D$ , which is only polynomial in  $K$ .
This conclusion has been drawn before (for example: Goodfellow et al., 2016, p. 370), but my argument makes the conditions more precise.
and not just those involving named entities.
A second option is to define a vector space with a logical interpretation. Grefenstette (2013) gives a logical interpretation to the type-driven tensorial framework (see §2), where the sentence space models truth values, and the noun space models a domain of  $N$  entities. However, Grefenstette shows that quantification would be nonlinear, so cannot be expressed using tensor contraction. Hedges and Sadrzadeh (2019) provide an alternative account which can deal with quantifiers, but at the expense of noun dimensions corresponding to sets of entities, so we have  $2^{N}$  dimensions for  $N$  entities.
Copestake and Herbelot (2012) propose that dimensions could correspond to logical expressions being true of an entity in a situation. However, this requires generalising from an actual distribution (based on observed utterances) to an ideal distribution (based on truth of logical expressions). They do not propose a concrete algorithm, but they discuss several challenges, and suggest that grounded data might be necessary. In this vein, Kuzmenko and Herbelot (2019) use the Visual Genome dataset (Krishna et al., 2017) to learn vector representations with logically interpretable dimensions, although these vectors are not as expressive as Copestake and Herbelot's ideal distributions.
Finally, a third option is to learn logical representations instead of vectors. For example, in my own work I have represented words as truth-conditional functions that are compatible with first-order logic (Emerson and Copestake, 2017b; Emerson, 2020b). Since referents are not observed in distributional semantics, this introduces latent variables that make the model computationally expensive, although there are ways to mitigate this (Emerson, 2020a).
Despite the computational challenges, I believe the right approach is to learn a logically interpretable model, either by defining a vector space with logical structure, or by directly using logical representations. However, an important question is what kind of logic to use. I argued in §4.1 that probabilities of truth and fuzzy truth values can capture vagueness, and there are corresponding logics.
In probability logic, propositions have probabilities of being true or false, with a joint distribution for the truth values of all propositions (for an introduction, see: Adams, 1998; Demey et al., 2013). In fuzzy logic, propositions have fuzzy truth values, and classical logical operators (such as:  $\land$ ,  $\lor$ ,  $\neg$ ) are replaced with fuzzy versions (for an introduc
tion, see: Hajek, 1998; Cintula et al., 2017). Fuzzy operators act directly on truth values – for example, given the fuzzy truth values of  $p$  and  $q$ , we can calculate the fuzzy truth value of  $p \lor q$ . In contrast, in probability logic, given probabilities of truth for  $p$  and  $q$ , we cannot calculate the probability of truth for  $p \lor q$ , unless we know the joint distribution.
A problem with fuzzy logic, observed by Fine (1975), comes with propositions like  $p \lor \neg p$ . For example, suppose we have a reddish orange object, so the truth of red and orange are both below 1. Intuitively, both red or not red and red or orange should definitely be true. However, in fuzzy logic, they could have truth below 1. This makes probability logic more appealing than fuzzy logic.[19]
Furthermore, there are well-developed frameworks for probabilistic logical semantics (for example: Goodman and Lassiter, 2015; Cooper et al., 2015), which a probabilistic distributional semantics could connect to, or draw inspiration from.
5.3 Context Dependence
The flipside of compositionality is context dependence: the meaning of an expression often depends on the context it occurs in. For example, a small elephant is not a small animal, but a large mouse is – the meanings of small and large depend on the nouns they modify. One goal for a semantic model is to capture how meaning depends on context.[20]
Following Recanati (2012), we can distinguish standing meaning, the context-independent meaning of an expression, and occasion meaning, the context-dependent meaning of an expression in a particular occasion of use.[21] However, every usage occurs in some context, so a standing meaning must be seen as an abstraction across usages, rather than a usage in a "null" context (for discussion, see: Searle, 1980; Elman, 2009).
One approach is to treat a distributional vector as a standing meaning, and modify it to produce occasion meanings. For example, vectors could be modified according to syntactic or semantic dependencies (for example: Erk and Padó, 2008; Thater et al., 2011; Dinu et al., 2012), or even chains of
<sup>19</sup>Hajek et al. (1995) prove that fuzzy logic can be used to provide upper and lower bounds on probabilities in a probability logic, giving it a different motivation.
20 Ultimately, this must include dependence on real-world context. Even the intuitive conclusion that a large mouse is a small animal depends on the implicit assumption that you and I are both humans, or at least, human-sized. From the perspective of an ant, a mouse is large animal.
This terminology adapts Quine (1960).
7443
dependencies (for example: Weir et al., 2016).
This mapping from standing vectors to occasion vectors can also be trained (for example: Czarnowska et al., 2019; Popa et al., 2019). Large language models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) can also be interpreted like this - these models map a sequence of input embeddings to a sequence of contextualised embeddings, which can be seen as standing meanings and occasion meanings, respectively.
Alternatively, standing meanings and occasion meanings can be represented by different kinds of object. Erk and Padó (2010) represent a standing meaning as a set of vectors (each derived from a single sentence of the training corpus), and an occasion meaning is a weighted sum of these vectors.
For a probabilistic model, calculating an occasion meaning can be cast as Bayesian inference, conditioning on the context. This gives us a well-understood theoretical framework, making it easier to generalise a model to other kinds of context.
Dinu and Lapata (2010) interpret a vector as a distribution over latent senses, where each component is the probability of a sense. Given probabilities of generating context words from latent senses, we can then condition the standing distribution on the context. However this model relies on a finite sense inventory, which I argued against in §4.2.
Lui et al. (2012) and Lau et al. (2012, 2014) use LDA (Blei et al., 2003), where an occasion meaning is a distribution over context words (varying continuously as topic mixtures), and a standing meaning is a prior over such distributions.[22] A separate model is trained for each target word. Chang et al. (2014) add a generative layer, allowing them to train a single model for all target words. However, a single sense is chosen in each context, giving a finite sense inventory.
Skip-gram can be interpreted as generating context words from a target word. While we can see an embedding as a standing meaning, nothing can be seen as an occasion meaning. Bražinskas et al. (2018) add a generative layer, generating a latent vector from the target word, then generating context words from this vector. We can see a latent vector as an occasion meaning, and a word's distribution over latent vectors as a standing meaning.
Finally, in my own work, I have also calculated
occasion meanings by conditioning on the context (Emerson and Copestake, 2017b), but in contrast to the above approaches, standing meanings are truth-conditional functions (binary classifiers), which I have argued for elsewhere in this paper.
A common thread among all of the above sections is that reaching our semantic goals requires structure beyond representing meaning as a point in space. In particular, it seems desirable to represent the meaning of a word as a region of space or as a classifier, and to work with probability logic.
However, there is a trade-off between expressiveness and learnability: the more structure we add, the more difficult it can be to work with our representations. To this end, there are promising neural architectures for working with structured data, such dependency graphs (for example: Marcheggiani and Titov, 2017) or logical propositions (for example: Rocktäschel and Riedel, 2017; Minervini et al., 2018). To mitigate computationally expensive calculations in probabilistic models, there are promising new techniques such as amortised variational inference, used in the Variational Autoencoder (Kingma and Welling, 2014; Rezende et al., 2014; Titsias and Lázaro-Gredilla, 2014).
My own recent work in this direction has been to develop the Pixie Autoencoder (Emerson, 2020a), and I look forward to seeing alternative approaches from other authors, as the field of distributional semantics continues to grow. I hope that this survey paper will help other researchers to develop the field in a way that keeps long-term goals in mind.
This paper is based on chapter 2 of my PhD thesis (Emerson, 2018). For invaluable advice on the structure and framing of that chapter, and therefore also of this paper, I want to thank my PhD supervisor Ann Copestake. I would also like to thank my PhD examiners, Katrin Erk and Paula Buttery, for feedback on that chapter, as well as Emily M. Bender, Guy Aglionby, Andrew Caines, and the NLIP reading group in Cambridge, for feedback on earlier drafts of this paper. I would like to thank ACL reviewers 1 & 3 for pointing out areas that were unclear, and reviewer 2 for their kind praise.
I am supported by a Research Fellowship at Gonville & Caius College, Cambridge.
22There are two distinct uses of a distribution here: to represent uncertainty, and to represent meaning. A sense is a topic mixture, parametrising a distribution over words; uncertainty is a Dirichlet distribution over topic mixtures.
Ernest W. Adams. 1998. A Primer of Probability Logic. Number 68 in CSLI Lecture Notes. Center for the Study of Language and Information (CSLI) Publications.
Keith Allan. 2001. Natural language semantics. Blackwell.
Ben Athiwaratkun and Andrew Gordon Wilson. 2017. Multimodal word distributions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), Long Papers, pages 1645-1656.
Ben Athiwaratkun and Andrew Gordon Wilson. 2018. Hierarchical density order embeddings. In Proceedings of the 6th International Conference on Learning Representations (ICLR).
Esma Balkir. 2014. Using density matrices in a compositional distributional model of meaning. Master's thesis, University of Oxford.
Esma Balkir, Mehrnoosh Sadrzadeh, and Bob Coecke. 2015. Distributional sentence entailment using density matrices. In Proceedings of the 1st International Conference on Topics in Theoretical Computer Science (TTCS), pages 1-22. International Federation for Information Processing (IFIP).
Dea Bankova, Bob Coecke, Martha Lewis, and Dan Marsden. 2019. Graded hyponymy for compositional distributional semantics. Journal of Language Modelling, 6(2):225-260.
Oren Barkan. 2017. Bayesian neural word embedding. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 3135-3143. Association for the Advancement of Artificial Intelligence.
Marco Baroni. 2016. Grounding distributional semantics in the visual world. *Language and Linguistics Compass*, 10(1):3-13.
Marco Baroni, Raffaela Bernardi, and Roberto Zamparelli. 2014. Frege in space: A program of compositional distributional semantics. Linguistic Issues in Language Technology (LiLT), 9.
Marco Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of the 15th Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1183-1193. Association for Computational Linguistics.
Lawrence W. Barsalou, Léo Dutraux, and Christoph Scheepers. 2018. Moving beyond the distinction between concrete and abstract concepts. Philosophical Transactions of the Royal Society B, 373(1752):20170144.