SlowGuess/ACL_2020_OCR huggingface dataset:
Guy Emerson  
Department of Computer Science and Technology  
University of Cambridge  
gete2@cam.ac.uk
Functional Distributional Semantics provides a linguistically interpretable framework for distributional semantics, by representing the meaning of a word as a function (a binary classifier), instead of a vector. However, the large number of latent variables means that inference is computationally expensive, and training a model is therefore slow to converge. In this paper, I introduce the Pixie Autoencoder, which augments the generative model of Functional Distributional Semantics with a graphconvolutional neural network to perform amortised variational inference. This allows the model to be trained more effectively, achieving better results on two tasks (semantic similarity in context and semantic composition), and outperforming BERT, a large pre-trained language model.
The aim of distributional semantics is to learn the meanings of words from a corpus (Harris, 1954; Firth, 1951, 1957). Many approaches learn a vector for each word, including count models and embedding models (for an overview, see: Erk, 2012; Clark, 2015), and some recent approaches learn a vector for each token in a particular context (for example: Peters et al., 2018; Devlin et al., 2019).
However, such vector representations do not make a clear distinction between words and the things they refer to. This means that such models are challenging to interpret semantically. In contrast, Functional Distributional Semantics (Emerson and Copestake, 2016) aims to provide a framework which can be interpreted in terms of model theory, a standard approach to formal semantics.
Furthermore, this framework supports first-order logic, where quantifying over logical variables is replaced by marginalising out random variables (Emerson and Copestake, 2017b; Emerson, 2020b).
This connection to logic is a clear strength over vector-based models. Even the linguistically inspired tensor-based framework of Coecke et al. (2010) and Baroni et al. (2014) cannot model quantifiers, as shown by Grefenstette (2013).
However, the linguistic interpretability of Functional Distributional Semantics comes at a computational cost, with a high-dimensional latent variable for each token. Training a model by gradient descent requires performing Bayesian inference over these latent variables, which is intractable to calculate exactly. The main theoretical contribution of this paper is to present an amortised variational inference algorithm to infer these latent variables. This is done using a graph-convolutional network, as described in §3.
The main empirical contribution of this paper is to demonstrate that the resulting system, the Pixie Autoencoder, improves performance on two semantic tasks, as described in §4. I also present the first published results of applying a large language model (BERT) to these tasks, showing that results are sensitive to linguistic detail in how the model is applied. Despite being a smaller model trained on less data, the Pixie Autoencoder outperforms BERT on both tasks.
While the proposed inference network is designed for Functional Distributional Semantics, the proposed techniques should also be of wider interest. From a machine learning perspective, amortised variational inference with graph convolutions (§3.3) could be useful in other tasks where the input data is a graph, and the use of belief propagation to reduce variance (§3.4) could be useful for training other generative models. However, the most important contribution of this work is from a computational semantics perspective. This paper takes an important step towards truth-conditional distributional semantics, showing that truth-conditional functions can be efficiently learnt from a corpus.
3982
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3982-3995 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: An example model structure with 14 individuals. Subscripts distinguish individuals with identical features, but are otherwise arbitrary. The pepper predicate is true of individuals inside the orange line, but the positions of individuals are otherwise arbitrary.
2 Functional Distributional Semantics
In this section, I summarise previous work on Functional Distributional Semantics. I begin in §2.1 by introducing model-theoretic semantics, which motivates the form of the machine learning model. I then explain in §2.2 how the meaning of a word is represented as a binary classifier, and finally present the probabilistic graphical model in §2.3.
2.1 Model-Theoretic Semantics
The basic idea of model-theoretic semantics is to define meaning in terms of truth, relative to model structures. A model structure can be understood as a model of the world. In the simplest case, it consists of a set of individuals (also called entities), as illustrated in Fig. 1. The meaning of a content word is called a predicate, and is formalised as a truth-conditional function, which maps individuals to truth values (either truth or falsehood).
Because of this precisely defined notion of truth, model theory naturally supports logic, and has become a prominent approach to formal semantics. For example, if we know the truth-conditional functions for pepper and red, we can use first-order logic to calculate the truth of sentences like Some peppers are red, for model structures like Fig. 1.
For detailed expositions, see: Cann (1993); Allan (2001); Kamp and Reyle (2013).
2.2 Semantic Functions
Functional Distributional Semantics (Emerson and Copestake, 2016; Emerson, 2018) embeds model-theoretic semantics into a machine learning model. An individual is represented by a feature vector, called a pixie. $^{1}$  For example, all three red pepper individuals in Fig. 1 would be represented by the
same pixie, as they have the same features. A predicate is represented by a semantic function, which maps pixies to probabilities of truth. For example, the function for pepper should map the red pepper pixie to a probability close to 1. This can be seen in formal semantics as a truth-conditional function, and in a machine learning as a binary classifier.
This ties in with a view of concepts as abilities, as proposed in some schools of philosophy (for example: Dummett, 1976, 1978; Kenny, 2010; Sutton, 2015, 2017), and some schools of cognitive science (for example: Labov, 1973; McCloskey and Glucksberg, 1978; Murphy, 2002, pp. 1-3, 134-138; Zentall et al., 2002). In NLP, some authors have suggested representing concepts as classifiers, including Larsson (2013), working in the framework of Type Theory with Records (Cooper, 2005; Cooper et al., 2015). Similarly, Schlangen et al. (2016) and Zarrieß and Schlangen (2017a,b) train image classifiers using captioned images.
We can also view such a classifier as defining a region in the space, as argued for by Gärdenfors (2000, 2014). This idea is used for distributional semantics by Erk (2009a,b), for colour terms by McMahan and Stone (2015), and for knowledge base completion by Bouraoui et al. (2017).
For a broader survey motivating the use of classifiers to represent meaning, see: Emerson (2020a).
2.3 Probabilistic Graphical Model
To learn semantic functions in distributional semantics, Emerson and Copestake define a probabilistic graphical model that generates semantic dependency graphs, shown in Fig. 3. The basic idea is that an observed dependency graph is true of some unobserved situation comprising a number of individuals. Given a sembank (a corpus parsed into dependency graphs), the model can be trained unsupervised, to maximise the likelihood of generating the data. An example graph is shown in Fig. 2, which corresponds to sentences like Every picture tells a story or The story was told by a picture (note that only content words have nodes).
More precisely, given a graph topology (a dependency graph where the edges are labelled but the nodes are not), the model generates a predicate for each node. Rather than directly generating predicates, the model assumes that each predicate describes an unobserved individual. The model
$^{1}$ Terminology introduced by Emerson and Copestake (2017a). This provides a useful shorthand for "feature representation of an individual".
2This assumes a neo-Davidsonian approach to event semantics (Davidson, 1967; Parsons, 1990), where verbal predicates are true of event individuals. It also assumes that a plural
$$
\mathrm {p i c t u r e} \xleftarrow {\mathrm {A R G 1}} \mathrm {t e l l} \xrightarrow {\mathrm {A R G 2}} \mathrm {s t o r y}
$$
Figure 2: A dependency graph, which could be generated by Fig. 3. Such graphs are observed in training.
Figure 3: Probabilistic graphical model for Functional Distributional Semantics. Each node is a random variable. The plate (box in middle) denotes repeated nodes.
Top row: individuals represented by jointly distributed pixie-valued random variables  $X, Y, Z$ , in a space  $\mathcal{X}$ . This is modelled by a Cardinality Restricted Boltzmann Machine (CaRBM), matching the graph topology.
Middle row: for each individual, each predicate  $r$  in the vocabulary  $\mathcal{V}$  is randomly true (T) or false (⊥), according to the predicate's semantic function. Each function is modelled by a feedforward neural net.
Bottom row: for each individual, we randomly generate one predicate, out of all predicates true of the individual. Only these nodes are observed.
first generates a pixie to represent each individual, then generates a truth value for each individual and each predicate in the vocabulary, and finally generates a single predicate for each individual. The pixies and truth values can be seen as a probabilistic model structure, which supports a probabilistic first-order logic (Emerson and Copestake, 2017b; Emerson, 2020b). This is an important advantage over other approaches to distributional semantics.
A pixie is defined to be a sparse binary-valued vector, with  $D$  units (dimensions), of which exactly  $C$  are active (take the value 1). The joint distribution over pixies is defined by a Cardinality Restricted Boltzmann Machine (CaRBM) (Swersky et al., 2012), which controls how the active units of each pixie should co-occur with the active
units of other pixies in the same dependency graph.
A CaRBM is an energy-based model, meaning that the probability of a situation is proportional to the exponential of the negative energy of the situation. This is shown in (1), where  $s$  denotes a situation comprising a set of pixies with semantic dependencies between them, and  $E(s)$  denotes the energy. The energy is defined in (2), where  $x \xrightarrow{l} y$  denotes a dependency from pixie  $x$  to pixie  $y$  with label  $l$ . The CaRBM includes a weight matrix  $w^{(l)}$  for each label  $l$ . The entry  $w_{ij}^{(l)}$  controls how likely it is for units  $i$  and  $j$  to both be active, when linked by dependency  $l$ . Each graph topology has a corresponding CaRBM, but the weight matrices are shared across graph topologies. Normalising the distribution in (2) is intractable, as it requires summing over all possible  $s$ .
$$
\mathbb {P} (s) \propto \exp (- E (s)) \tag {1}
$$
$$
\mathbb {P} (s) \propto \exp \left(\sum_ {x \xrightarrow {l} y \text {i n} s} w _ {i j} ^ {(l)} x _ {i} y _ {j}\right) \qquad (2)
$$
The semantic function  $t^{(r)}$  for a predicate  $r$  is defined to be one-layer feedforward net, as shown in (3), where  $\sigma$  denotes the sigmoid function. Each predicate has a vector of weights  $v^{(r)}$ .
$$
t ^ {(r)} (x) = \sigma \left(v _ {i} ^ {(r)} x _ {i}\right) \tag {3}
$$
Lastly, the probability of generating a predicate  $r$  for a pixie  $x$  is given in (4). The more likely  $r$  is to be true, the more likely it is to be generated. Normalising requires summing over the vocabulary.
$$
\mathbb {P} (r \mid x) \propto t ^ {(r)} (x) \tag {4}
$$
In summary, the model has parameters  $w^{(l)}$  (the world model), and  $v^{(r)}$  (the lexical model). These are trained on a sembank using the gradients in (5), where  $g$  is a dependency graph. For  $w^{(l)}$ , only the first term is nonzero; for  $v^{(r)}$ , only the second term.
$$
\begin{array}{l} \frac {\partial}{\partial \theta} \log \mathbb {P} (g) = \left(\mathbb {E} _ {s | g} - \mathbb {E} _ {s}\right) \left[ \frac {\partial}{\partial \theta} (- E (s)) \right] \tag {5} \\ + \mathbb {E} _ {s | g} \left[ \frac {\partial}{\partial \theta} \log \mathbb {P} (g \mid s) \right] \\ \end{array}
$$
noun corresponds to a plural individual, which would be compatible with Link (1983)'s approach to plural semantics.
<sup>3</sup>Although a pixie is a feature vector, the features are all latent in distributional semantics, in common with models like LDA (Blei et al., 2003) or Skip-gram (Mikolov et al., 2013).
$^4$ I follow the Einstein summation convention, where a repeated subscript is assumed to be summed over. For example,  $x_i y_i$  is a dot product. Furthermore, I use uppercase for random variables, and lowercase for values. I abbreviate  $\mathbb{P}(X = x)$  as  $\mathbb{P}(x)$ , and I abbreviate  $\mathbb{P}(T_{r,X} = \top)$  as  $\mathbb{P}(t_{r,X})$ .
3 The Pixie Autoencoder
A practical challenge for Functional Distributional Semantics is training a model in the presence of high-dimensional latent variables. In this section, I present the Pixie Autoencoder, which augments the generative model with an encoder that predicts these latent variables.
For example, consider dependency graphs for The child cut the cake and The gardener cut the grass. These are true of rather different situations. Although the same verb is used in each, the pixie for cut should be different, because they describe events with different physical actions and different tools (slicing with a knife vs. driving a lawn mower). Training requires inferring posterior distributions for these pixies, but exact inference is intractable.
In §3.1 and §3.2, I describe previous work: amortised variational inference is useful to efficiently predict latent variables; graph convolutions are useful when the input is a graph. In §3.3, I present the encoder network, to predict latent pixies in Functional Distributional Semantics. It uses the tools introduced in §3.1 and §3.2, but modified to better suit the task. In §3.4, I explain how the encoder network can be used to train the generative model, since training requires the latent variables. Finally, I summarise the architecture in §3.5, and compare it to other autoencoders in §3.6.
3.1 Amortised Variational Inference
Calculating the gradients in (5) requires taking expectations over situations (both the marginal expectation  $\mathbb{E}_s$ , and the conditional expectation  $\mathbb{E}_{s|g}$  given a graph). Exact inference would require summing over all possible situations, which is intractable for a high-dimensional space.
This is a general problem when working with probabilistic models. Given an intractable distribution  $\mathbb{P}(x)$ , a variational inference algorithm approximates this by a simpler distribution  $\mathbb{Q}(x)$ , parametrised by  $q$ , and then optimises the parameters so that  $\mathbb{Q}$  is as close as possible to  $\mathbb{P}$ , where closeness is defined using KL-divergence (for a detailed introduction, see: Jordan et al., 1999).
However, variational inference algorithms typically require many update steps in order to optimise the approximating distribution  $\mathbb{Q}$ . An amortised variational inference algorithm makes a further approximation, by estimating the parameters  $q$  using an inference network (Kingma and Welling, 2014; Rezende et al., 2014; Titsias and Lazaro-Gredilla,
2014). The inference network might not predict the optimal parameters, but the calculation can be performed efficiently, rather than requiring many update steps. The network has its own parameters  $\phi$ , which are optimised so that it makes good predictions for the variational parameters  $q$ .
3.2 Graph Convolutions
For graph-structured input data, a standard feedforward neural net is not suitable. In order to share parameters across similar graph topologies, an appropriate architecture is a graph-convolutional network (Duvenaud et al., 2015; Kearnes et al., 2016; Kipf and Welling, 2017; Gilmer et al., 2017). This produces a vector representation for each node in the graph, calculated through a number of layers. The vector for a node in layer  $k$  is calculated based only on the vectors in layer  $k - 1$  for that node and the nodes connected to it. The same weights are used for every node in the graph, allowing the network to be applied to different graph topologies.
For linguistic dependency graphs, the dependency labels carry important information. Marcheggiani and Titov (2017) propose using a different weight matrix for each label in each direction. This is shown in (6), where:  $h^{(k,X)}$  denotes the vector representation of node  $X$  in layer  $k$ ;  $w^{(k,l)}$  denotes the weight matrix for dependency label  $l$  in layer  $k$ ;  $f$  is a non-linear activation function; and the sums are over outgoing and incoming dependencies. There is a separate weight matrix  $w^{(k,l^{-1})}$  for a dependency in the opposite direction, and as well as a matrix  $w^{(k,\mathrm{self})}$  for updating a node based on itself. Bias terms are not shown.
$$
\begin{array}{l} h _ {i} ^ {(k, X)} = f \left(w _ {i j} ^ {(k, \text {s e l f})} h _ {j} ^ {(k - 1, X)} \right. \\ + \sum_ {Y \stackrel {l} {\leftarrow} X} w _ {i j} ^ {(k, l)} h _ {j} ^ {(k - 1, Y)} \tag {6} \\ \left. + \sum_ {Y \xrightarrow {l} X} w _ {i j} ^ {(k, l ^ {- 1})} h _ {j} ^ {(k - 1, Y)}\right) \\ \end{array}
$$
3.3 Predicting Pixies
For Functional Distributional Semantics, Emerson and Copestake (2017a) propose a mean-field variational inference algorithm, where  $\mathbb{Q}$  has an independent probability  $q_{i}^{(X)}$  of each unit  $i$  being active, for each node  $X$ . Each probability is optimised based on the mean activation of all other units.
For consistency with Fig. 3, I write  $X$  for a node (a random variable), rather than  $x$  (a pixie).
Figure 4: Graph-convolutional inference network for Fig. 3. The aim is to predict the posterior distribution over the pixie nodes  $X, Y, Z$ , given the observed predicates  $p, q, r$ . Each edge indicates the weight matrix used in the graph convolution, as defined in (6). In the bottom row, the input at each node is an embedding for the node's predicate. The intermediate representations  $h$  do not directly correspond to any random variables in Fig. 3. Conversely, the truth-valued random variables in Fig. 3 are not directly represented here.
This makes the simplifying assumption that the posterior distribution can be approximated as a single situation with some uncertainty in each dimension. For example, for a dependency graph for The gardener cut the grass, three mean vectors are inferred, for the gardener, the cutting event, and the grass. These vectors are "contextualised", because they are jointly inferred based on the whole graph.
I propose using a graph-convolutional network to amortise the inference of the variational mean-field vectors. In particular, I use the formulation in (6), with two layers. The first layer has a tanh activation, and the second layer has a sigmoid (to output probabilities). In addition, if the total activation in the second layer is above the total cardinality  $C$ , the activations are normalised to sum to  $C$ . The network architecture is illustrated in Fig. 4.
The network is trained to minimise the KL-divergence from  $\mathbb{P}(s|g)$  (defined by the generative model) to  $\mathbb{Q}(s)$  (defined by network's output). This is shown in (7), where  $\mathbb{E}_{\mathbb{Q}(s)}$  denotes an expectation over  $s$  under the variational distribution.
$$
D (\mathbb {Q} \| \mathbb {P}) = - \mathbb {E} _ {\mathbb {Q} (s)} \left[ \log \left(\frac {\mathbb {P} (s \mid g)}{\mathbb {Q} (s)}\right) \right] \tag {7}
$$
To minimise the KL-divergence, we can differen
tiate with respect to the inference network parameters  $\phi$ . This gives (8), where  $H$  denotes entropy.
$$
\begin{array}{l} \frac {\partial}{\partial \phi} D (\mathbb {Q} \| \mathbb {P}) = - \frac {\partial}{\partial \phi} \mathbb {E} _ {\mathbb {Q} (s)} [ \log \mathbb {P} (s) ] \\ - \frac {\partial}{\partial \phi} \mathbb {E} _ {\mathbb {Q} (s)} [ \log \mathbb {P} (g \mid s) ] \tag {8} \\ - \frac {\partial}{\partial \phi} H (\mathbb {Q}) \\ \end{array}
$$
The first term can be calculated exactly, because the log probability is proportional to the negative energy, which is a linear function of each pixie, and the normalisation constant is independent of  $s$  and  $\mathbb{Q}$ . This term therefore simplifies to the energy of the mean-field pixies,  $\frac{\partial}{\partial\phi} E(\mathbb{E}[s])$ .
The last term can be calculated exactly, because  $\mathbb{Q}$  was chosen to be simple. Since each dimension is independent, it is  $\sum_{q} q \log q + (1 - q) \log (1 - q)$ , summing over the variational parameters.
The second term is more difficult, for two reasons. Firstly, calculating the probability of generating a predicate requires summing over all predicates, which is computationally expensive. We can instead sum over a random sample of predicates (along with the observed predicate). However, by ignoring most of the vocabulary, this will overestimate the probability of generating the correct predicate. I have mitigated this by upweighting this term, similarly to a  $\beta$ -VAE (Higgins et al., 2017).
The second problem is that the log probability of a predicate being true is not a linear function of the pixie. The first-order approximation would be to apply the semantic function to the mean-field pixie, as suggested by Emerson and Copestake (2017a). However, this is a poor approximation when the distribution over pixies has high variance. By approximating a sigmoid using a probit and assuming the input is approximately Gaussian, we can derive (9) (Murphy, 2012, §8.4.4.2). Intuitively, the higher the variance, the closer the expected value to  $\frac{1}{2}$ . For a Bernoulli distribution with probability  $q$ , scaled by a weight  $v$ , the variance is  $v^2 q(1 - q)$ .
$$
\mathbb {E} [ \sigma (x) ] \approx \sigma \left(\frac {\mathbb {E} [ x ]}{\sqrt {1 + \frac {\pi}{8} \operatorname {V a r} [ x ]}}\right) \tag {9}
$$
With the above approximations, we can calculate (4) efficiently. However, because the distribution over predicates in (4) only depends on relative probabilities of truth, the model might learn to keep them all close to 0, which would damage the logical interpretation of the model. To avoid this, I
have modified the second term of (5) and second term of (8), using not only the probability of generating a predicate for a pixie,  $\mathbb{P}(r\mid x)$ , but also the probability of the truth of a predicate,  $\mathbb{P}(t_{r,X}\mid x)$ . This technique of constraining latent variables to improve interpretability is similar to how Rei and Søgaard (2018) constrain attention weights.
Finally, as with other autoencoder models, there is a danger of learning an identity function that generalises poorly. Here, the problem is that the pixie distribution for a node might be predicted based purely on the observed predicate for that node, ignoring the wider context. To avoid this problem, we can use dropout on the input, a technique which has been effective for other NLP models (Iyyer et al., 2015; Bowman et al., 2016), and which is closely related to denoising autoencoders (Vincent et al., 2008). More precisely, we can keep the graph topology intact, but randomly mask out the predicates for some nodes. For a masked node  $X$ , I have initialised the encoder with an embedding as shown in (10), which depends on the node's dependencies (only on the label of each dependency, not on the predicate of the other node).
$$
e ^ {(X)} = e ^ {(\text {d r o p})} + \sum_ {Y \stackrel {l} {\leftarrow} X} e ^ {(\text {d r o p}, l)} + \sum_ {Y \stackrel {l} {\rightarrow} X} e ^ {(\text {d r o p}, l ^ {- 1})} \tag {10}
$$
3.4 Approximating the Prior Expectation