SlowGuess/ACL_2020_OCR huggingface dataset:
Guy Emerson
Department of Computer Science and Technology
gete2@cam.ac.uk
Distributional semantic models have become a mainstay in NLP, providing useful features for downstream tasks. However, assessing long-term progress requires explicit long-term goals. In this paper, I take a broad linguistic perspective, looking at how well current models can deal with various semantic challenges. Given stark differences between models proposed in different subfields, a broad perspective is needed to see how we could integrate them. I conclude that, while linguistic insights can guide the design of model architectures, future progress will require balancing the often conflicting demands of linguistic expressiveness and computational tractability.
In order to assess progress in any field, the goals need to be clear. In assessing progress in semantics, Koller (2016) contrasts "top-down" and "bottom-up" approaches: a top-down approach begins with an overarching goal, and tries to build a model to reach it; a bottom-up approach begins with existing models, and tries to extend them towards new goals.<sup>1</sup> Like much of NLP, distributional semantics is largely bottom-up: the goals are usually to improve performance on particular tasks, or particular datasets. Aiming to improve NLP applications is of course a legitimate decision, but Koller points out a problem if there is no top-down goal: "Bottom-up theories are intrinsically unfalsifiable... We won't know where distributional semantics is going until it has a top-down element". This is contrasted against truth-conditional semantics, a traditional linguistic approach which is largely top-down: "truth-conditional semantics hasn't reached its goal, but at least we knew what the goal was".
In this paper, I take a long-term linguistic perspective, where the top-down goal is to characterise
the meanings of all utterances in a language. This is an ambitious goal, and a broad one. To make this goal more precise, in the following sections I will elaborate on several aspects of meaning which could be considered crucial. For each aspect, I identify a plausible goal, lay out out the space of possible models, place existing work in this space, and evaluate which approaches seem most promising. By making the goals explicit, we can assess whether we are heading in the right direction, and we can assess what still needs to be done. If a reader should disagree with my conclusions, they should start by looking at my goals.
2 Background: Distributional Semantics
The aim of distributional semantics is to learn the meanings of linguistic expressions from a corpus of text. The core idea, known as the distributional hypothesis, is that the contexts in which an expression appears give us information about its meaning.[2]
The idea has roots in American structuralism (Harris, 1954) and British lexicology (Firth, 1951,  $1957)^{3}$ , and with the advent of modern computing, it began to be used in practice. In a notable early work, Spärck-Jones (1964) represented word meanings as boolean vectors, based on a thesaurus.
Distributional semantics has become widespread in NLP, first with the rise of count vectors (for an overview, see: Erk, 2012; Clark, 2015), then of word embeddings (Mikolov et al., 2013), and most recently, of contextualised embeddings (Peters et al., 2018; Devlin et al., 2019). What all of these approaches share is that they learn representations in an unsupervised manner on a corpus.
2The hypothesis is often stated more narrowly, to say that similar words appear in similar contexts, but in this paper I am interested in semantics beyond just similarity.
$^{3}$ Firth used the term collocational, not distributional.
4For connections between count vectors and embeddings, see: Levy and Goldberg (2014); Cotterell et al. (2017); for connections with contextual embeddings: Kong et al. (2020).
For further discussion, see: Bender and Koller (2020).
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7436-7453 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
While much work takes a bottom-up approach, as Koller observes, a notable exception is the typedriven tensorial framework of Coecke et al. (2010) and Baroni et al. (2014), which has broad linguistic goals, and will be mentioned in several sections below. This framework represents the meanings of words as tensors, and constructs phrase meanings using tensor contraction based on predicateargument structure. For example, there is one vector space for nouns, and a second vector space for sentences, so intransitive verbs are matrices (mapping noun vectors to sentence vectors).
3 Meaning and the World
Language is always about something. In this section, I discuss challenges in connecting a semantic model to things in the world.
3.1 Grounding
As Harnad (1990) discusses, if the meanings of words are defined only in terms of other words, these definitions are circular. One goal for a semantic model is to capture how language relates to the world, including sensory perception and motor control – this process of connecting language to the world is called grounding.[5]
A purely distributional model is not grounded, as it is only trained on text, with no direct link to the world. There are several ways we could try to ground a distributional model (for an overview, see: Baroni, 2016). The simplest way is to train a distributional model as normal, then combine it with a grounded model. For example, Bruni et al. (2011) concatenate distributional vectors and image feature vectors. This has also been applied to other senses: Kiela et al. (2015) use olfactory data, and Kiela and Clark (2017) use both visual and auditory data. However, while there is grounded information in the sensory dimensions, concatenation leaves the distributional dimensions ungrounded.
A second approach is to find correlations between distributional and sensory features. For example, Bruni et al. (2014) perform SVD on concatenated vectors, Silberer and Lapata (2014) train an autoencoder on concatenated vectors, and Lazari-dou et al. (2014) and Bulat et al. (2016) learn a mapping from distributional vectors to visual vectors (and vice versa). However, there is no guarantee
that every distributional feature will correlate with sensory features. Distributional features without correlations will remain ungrounded.
Finally, a third approach is joint learning – we define a single model, whose parameters are learnt based on both corpus data and grounded data. For example, Feng and Lapata (2010) train an LDA model (Blei et al., 2003) for both words and “visual words” (clusters of visual features). Lazaridou et al. (2015) use a Skip-gram model (Mikolov et al., 2013) to jointly predict both words and images. Kiros et al. (2014) embed both text and images in a single space, training an RNN to process captions, and a CNN to process images. Pure distributional models look for word co-occurrence patterns, while joint models prefer co-occurrence patterns that match the grounded data. For this reason, I believe joint learning is the right approach to ground corpus data – semantic representations can be connected to grounded data from the outset, rather than trying to make such connections after the fact.
However, we must still make sure that all distributional features are grounded. With Feng and Lapata's LDA model, some topics might only generate words rather than "visual words". Similarly, with Lazaridou et al.'s joint Skip-gram model, some embeddings might only predict words rather than images. Conversely, we also need to make sure that we make full use of corpus data, rather than discarding what is difficult to ground. For example, Kiros et al.'s joint embedding model learns sentence embeddings in order to match them to images. It is not obvious how this approach could be extended so that we can learn embeddings for sentences that cannot be easily depicted in an image.
This leads to the question: how should a joint architecture be designed, so that we can fully learn from corpus data, while ensuring that representations are fully grounded? Grounding is hard, and indeed Kuhnle et al. (2018) find that some semantic constructions (such as superlatives) are much harder for grounded models to learn than others. In the following section, I discuss how language relates to the world. Clarifying this relationship should help us to design good joint architectures.
3.2 Concepts and Referents
How do meanings relate to the world? In truth-conditional semantics, the answer is that meaning is defined in terms of truth. If an agent under
5This includes connecting abstract concepts to the world, although such connections are necessarily more indirect. For further discussion, see: Blondin-Masse et al. (2008); Pecher et al. (2011); Pulvermüller (2013); Barsalou et al. (2018)
For a discussion of this point, see: Lewis (1970). For an
stands a language, then in any given situation, they know how to evaluate whether a sentence is true or false of that situation. An advantage of this approach is that it supports logical reasoning, which I will discuss in §5.2. One goal for a semantic theory is to be able to generalise to new situations. This is difficult for traditional truth-conditional semantics, with classical theories challenged on both philosophical grounds (for example: Wittgenstein, 1953, §66–71) and empirical grounds (for example: Rosch, 1975, 1978). However, a machine learning approach seems promising, since generalising to new data is a central aim of machine learning.
For a semantic model to be compatible with truth-conditional semantics, it is necessary to distinguish a concept (the meaning of a word) from a referent (an entity the word can refer to). The importance of this distinction has been noted for some time (for example: Ogden and Richards, 1923). A concept's set of referents is called its extension.
Even if we can construct grounded concept vectors, as discussed in §3.1, there is still the question of how to relate a concept vector to its referents. $^{10}$  One option is to embed both concepts and entities in the same space. We then need a way to decide how close the vectors need to be, for the entity to be in the concept's extension. A second option is to embed concepts and referents in distinct spaces. We then need a way to relate the two spaces.
In both cases, we need additional structure beyond representing concepts and referents as points. One solution is to represent a concept by a region of space (Gärdenfors, 2000, 2014). Entities embedded inside the region are referents, while those outside are not. For example, McMahan and Stone (2015) learn representations of colour terms, which are grounded in a well-understood perceptual space.
A related idea is to represent a concept as a binary classifier, where an entity is the input.11 One class is the concept's extension, and the other class
is everything else. Larsson (2013) represents the meaning of a perceptual concept as a classifier of perceptual input. A number of authors have trained image classifiers using captioned images (for example: Schlangen et al., 2016; Zarrieß and Schlangen, 2017a,b; Utescher, 2019; Matsson et al., 2019).
Such representations have however seen limited use in distributional semantics. Erk (2009a,b) and Dong et al. (2018) learn regions, but relying on pre-trained vectors, which may have already lost referential information (such as co-reference) that we would like to capture. Jameel and Schockaert (2017) learn a hybrid model, where each word is represented by a point (as a target word) and a region (as a context word). In my own work, I have learnt classifiers (Emerson and Copestake, 2016, 2017a,b), but with a computationally expensive model that is difficult to train. The computational challenge is partially resolved in my most recent work (Emerson, 2020a), but there is still work to be done in scaling up the model to make full use of the corpus data. The best way to design such a model, so that it can both make full use of the data and can be trained efficiently, is an open question.
4 Lexical Meaning
In this section, I discuss challenges in representing the meanings of individual words.
4.1 Vagueness
Entities often fall along a continuum without a sharp cutoff between concepts. This is called vagueness (or gradedness). (For an overview, see: Sutton, 2013, chapter 1; Van Deemter, 2010.) For example, Labov (1973) investigated the boundaries between concepts like cup, mug, and bowl, asking participants to name drawings of objects. For typical referents, terms were used consistently; meanwhile, for objects that were intermediate between concepts (for example, something wide for a cup but narrow for a bowl), terms were used inconsistently. For these borderline cases, a single person may make different judgements at different times (McCloskey and Glucksberg, 1978). One goal for a semantic model is to capture how it can be unclear whether an entity is an referent of a concept.
One approach is to use fuzzy truth values, which are not binary true/false, but rather values in the range [0,1], where 0 is definitely false, 1 is definitely true, and intermediate values represent borderline cases (Zadeh, 1965, 1975). Fuzzy logic has
introduction to truth-conditional semantics, see: Cann (1993); Allan (2001); Kamp and Reyle (2013).
7On the notion of situation, see: Barwise and Perry (1983). On knowing how to evaluate truth values vs. actually evaluating truth values, see: Dummett (1976, 1978).
$^{8}$ Following Murphy (2002, pp. 4-5), I use the term concept without committing to a particular theory of concepts.
$9$ Or denotation. In psychology, the term category is also used (for example: Smith and Medin, 1981; Murphy, 2002).
10While distributional representations can be learnt for named entities (for example: Herbelot, 2015; Boleda et al., 2017), most real-world entities are not mentioned in text.
11For deterministic regions and classifiers, there is a one-to-one mapping between them, but this is not true for probabilistic regions and classifiers, due to covariance.
not seen much use in computational linguistics. $^{12}$
A second solution is to stick with binary truth values, but using probability theory to formalise uncertainty about truth, as has been proposed in formal semantics (for example: Lassiter, 2011; Fernandez and Larsson, 2014; Sutton, 2015, 2017). At the level of a single concept, there is not much to decide between fuzzy and probabilistic accounts, since both assign values in the range [0,1]. However, we will see in §5.2 that they behave differently at the level of sentences.
Uncertainty has also been incorporated into distributional vector space models. Vilnis and McCallum (2015) extend Mikolov et al.'s Skip-gram model, representing meanings as Gaussian distributions over vectors. Barkan (2017) incorporate uncertainty into Skip-gram using Bayesian inference – rather than optimising word vectors, the aim is to calculate the posterior distribution over word vectors, given the observed data. The posterior is approximated as a Gaussian, so these two approaches produce the same kind of object. Balkir (2014), working within the type-driven tensorial framework (see §2), uses a quantum mechanical “mixed state” to model uncertainty in a tensor. For example, this replaces vectors by matrices, and replaces matrices by fourth-order tensors.
While these approaches represent uncertainty, it is challenging to use them to capture vagueness. This basic problem is this: a distribution allows us to generate referents of a concept, but how can we go in the other direction, to recognise referents of a concept? It is tempting to classify a point using the probability density at that point, but if we compare a more general term with a more specific term (like animal and dog), we find a problem: a more general term has its probability mass spread more thinly, and hence has a lower probability density than the more specific term, even if both terms could be considered true. I argued in §3.2 that, to talk about truth, we need to represent predicates as regions of space or as classifiers. While a distribution over a space might at first sight look like a region of space, normalising the probability mass to sum to 1 makes a distribution a different kind of object.
4.2 Polysemy
The meaning of a word can often be broken up into distinct senses. Related senses are called polysemous: for example, school can refer to a building or an institution. In contrast, homonymous senses are unrelated: for example, a school of fish. All of the above senses of school are also lexicalised - established uses that a speaker would have committed to memory, rather than inferring from context. I will discuss context-dependent meaning in §5.3, and focus here on lexicalised meaning. One goal for a semantic model is to capture how a word can have a range of polysemous senses.
One solution is to learn a separate representation for each sense (for example: Schütze, 1998; Rapp, 2004; Li and Jurafsky, 2015; for a survey, see: Camacho-Collados and Pilehvar, 2018). However, deciding on a discrete set of senses is difficult, and practical efforts at compiling dictionaries have not provided a solution. Indeed, the lexicographer Sue Atkins bluntly stated, "I don't believe in word senses".<sup>13</sup> Although the sense of a word varies across usages, there are many ways that we could cluster usages into a discrete set of senses, a point made by many authors (for example: Spärck-Jones, 1964; Kilgarriff, 1997, 2007; Hanks, 2000; Erk, 2010). To quantify this intuition, Erk et al. (2009, 2013) produced the WSsim and Usim datasets, where annotators judged the similarity between dictionary senses, and the similarity between individual usages, respectively. McCarthy et al. (2016) quantify "clusterability" in USim, showing that for some words, usages cannot be clustered into discrete senses. A good semantic model should therefore be able to capture variation in meaning without resorting to finite sense inventories.
We could instead learn a single representation for all polysemous senses together. Indeed, Ruhl (1989) argues that even frequent terms with many apparent senses, such as bear and hit, can be analysed as having a single underspecified meaning, with the apparent diversity of senses explainable from context. The challenge is then to represent such a meaning without overgeneralising to cases where the word wouldn't be used, and to model how meanings are specialised in context. The second half of this challenge will be discussed in §5.3.
I have already argued in previous sections that we should move away from representing each word as a single vector. As discussed in  $\S 4.1$ , words
$^{13}$ Kilgarriff (1997) and Hanks (2000) both quote Atkins.
$^{12}$ Carvalho et al. (2012) survey fuzzy logic in NLP, noting that its use is in decline, but they do not mention distributional semantics. Proposals such as Monte Carlo Semantics (Bergmair, 2010) and Fuzzy Natural Logic (Novák, 2017) do not provide an approach to distributional semantics. A rare exception is Runkler (2016), who infers fuzzy membership functions from pre-trained vectors.
7439
can be represented with distributions, and such an approach has also been applied to modelling word senses. For example, Athiwaratkun and Wilson (2017) use a mixture of Gaussians, extending Vilnis and McCallum's model to allow multiple senses. However, this ultimately models a fixed number of senses (one for each Gaussian). In principle, a distribution could be parametrised in a more general way, moving beyond finite mixture models. In the type-driven tensorial framework (see §2), Piedeleu et al. (2015) use mixed quantum states, similarly to Balkir's approach (see §4.1). Although they only propose this approach for homonymy, it could plausibly be used for polysemy as well.
If a word is represented by a region, or by a classifier, we don't have the problem of finite sense inventories, as long as the region or classifier is parametrised in a general enough way – for example, a multi-layer neural net classifier, rather than a finite mixture of simple classifiers.
4.3 Hyponymy
In the previous two sections, I discussed meanings of single words. However, words do not exist on their own, and one goal for semantic model is to represent relations between them. A classic relation is hyponymy, $^{14}$  which describes when one term (the hyperonym or hybernym) has a more general meaning than another (the hyponym). Words that share a hyperonym are called co-hyponyms.
In a vector space model, it is not clear how to say if one vector is more general than another. One idea is that a hyperonym should occur in all the contexts of its hyponyms. This is known as the Distributional Inclusion Hypothesis (DIH; Weeds et al., 2004; Geffet and Dagan, 2005). Using this idea and tools from information retrieval, Kotlerman et al. (2009, 2010) define the "balAPinc" measure of hyponymy. Herbelot and Ganesalingam (2013) view a vector as a distribution over contexts, using KL-divergence to measure hyponymy. Rei (2013) gives an overview of hyponymy measures, and proposes a weighted cosine measure. For embeddings, the motivation for such measures is less direct, but dimensions can be seen as combinations of contexts. Indeed, Rei and Briscoe (2014) find embeddings perform almost as well as count vectors.
However, a speaker is likely to choose an expression with a degree of generality appropriate for the discourse (the Maxim of Quantity; Grice, 1967), and hence the DIH can be questioned. Rimell (2014) points out that some contexts are highly specific. For example, mane is a likely context of lion but not animal, even though lion is a hyponym of animal, contradicting the DIH. Rimell instead proposes measuring hyponymy using coherence (formalised using pointwise mutual information): the contexts of a general term minus those of a hyponym are coherent, but the reverse is not true.
Moving away from count vectors and pre-trained embeddings, there are other options. One is to build the hyponymy relation into the definition of the space. For example, Vendrov et al. (2016) use nonnegative vectors, where one vector is a hyponym of another if it has a larger value in every dimension. They train a model on WordNet (Miller, 1995; Fellbaum, 1998). Building on this, Li et al. (2017) learn from both WordNet and text.
However, for a hierarchy like WordNet, there are exponentially more words lower down. This cannot be embedded in Euclidean space without words lower in the hierarchy being increasingly close together. Nickel and Kiela (2017) propose using hyperbolic space, where volume increases exponentially as we move away from any point. Tifrea et al. (2019) build on this, adapting Glove (Pennington et al., 2014) to learn hyperbolic embeddings from text. However, this approach does not generalise to non-tree hierarchies – for example, WordNet gives bass as a hyponym of singer, voice, melody, pitch, and instrument. Requiring that bass is represented close to all its hyperonyms also forces them close together (by the triangle inequality), which we may not want, since they are in distant parts of the hierarchy.
Alternatively, we can view hyponymy as classification, and simply use distributional vectors to provide input features (for example: Weeds et al., 2014; Rei et al., 2018). However, under this view, hyponymy is an opaque relationship, making it difficult to analyse why one vector is classified as a hyponym of another. Indeed, Levy et al. (2015) find that such classifiers mainly learn which words are common hyperonyms.
Moving away from vector representations, it can be easier to define hyponymy. Erk (2009a,b) and Gärdenfors (2014, §6.4) discuss how using regions of space provides a natural definition:  $P$  is a hy-
This is also referred to as lexical entailment, making a link with logic (see §5.2). Other relations include antonymy, meronymy, and selectional preferences. For reasons of space, I have decided to discuss one relation in detail, rather than many relations briefly. Hyponymy could be considered basic.
7440