SlowGuess/ACL_2020_OCR huggingface dataset:
The previous section explains the inference network and how it is trained. To train the generative model, the predictions of the inference network (without dropout) are used to approximate the conditional expectations  $\mathbb{E}_{s|g}$  in (5). However, the prior expectation  $\mathbb{E}_s$  cannot be calculated using the inference network. Intuitively, the prior distribution encodes a world model, and this cannot be summarised as a single mean-field situation.
Emerson and Copestake (2016) propose an MCMC algorithm using persistent particles, summing over samples to approximate the expectation. Many samples are required for a good approximation, which is computationally expensive. Taking a small number produces high variance gradients, which makes training less stable.
However, we can see in (5) that we don't need the prior expectation  $\mathbb{E}_s$  on its own, but rather the difference  $\mathbb{E}_{s|g} - \mathbb{E}_s$ . So, to reduce the variance of gradients, we can try to explore the prior distribution only in the vicinity of the inference network's predictions. In particular, I propose taking the inference network's predictions and updating
this mean-field distribution to bring it closer to the prior under the generative model. This can be done using belief propagation (for an introduction, see: Yedidia et al., 2003), as applied to CaRBMs by Swersky et al. (2012). For example, given the predicted mean-field vectors for a gardener cutting grass, we would modify these vectors to make the distribution more closely match what is plausible under the generative model (based on the world model, ignoring the observed predicates).
This can be seen as the bias-variance trade-off: the inference network introduces a bias, but reduces the variance, thereby making training more stable.
3.5 Summary
The Pixie Autoencoder is a combination of the generative model from Functional Distributional Semantics (generating dependency graphs from latent situations) and an inference network (inferring latent situations from dependency graphs), as illustrated in Figs. 3 and 4. They can be seen as an decoder and encoder, respectively.
It is trained on a sembank, with the generative model maximising the likelihood of the dependency graphs, and the inference network minimising KL-divergence with the generative model. To calculate gradients, the inference network is first applied to a dependency graph to infer the latent situation. The generative model gives the energy of the situation and the likelihood of the observed predicates (compared with random predicates). We also calculate the entropy of the situation, and apply belief propagation to get a situation closer to the prior. This gives us all terms in (5) and (8).
A strength of the Pixie Autoencoder is that it supports logical inference, following Emerson and Copestake (2017a). This is illustrated in Fig. 5. For example, for a gardener cutting grass or a child cutting a cake, we could ask whether the cutting event is also a slicing event or a mowing event.
3.6 Structural Prior
I have motivated the Pixie Autoencoder from the perspective of the generative model. However, we can also view it from the perspective of the encoder, comparing it with a Variational Autoencoder (VAE) which uses an RNN to generate text from a latent vector (Bowman et al., 2016). The VAE uses a Gaussian prior, but the Pixie Autoencoder has a structured prior defined by the world model.
Hoffman and Johnson (2016) find that VAEs struggle to fit a Gaussian prior. In contrast, the
Figure 5: An example of logical inference, building on Fig. 4. Given an observed semantic dependency graph (here, with three nodes, like Fig. 2, with predicates  $p$ ,  $q$ ,  $r$ ), we would like to know if some predicate is true of some latent individual (here, if  $a$  is true of  $Y$ ). We can apply the inference network to infer distributions for the pixie nodes, and then apply a semantic function to a pixie node (here, the function for  $a$  applied to  $Y$ ).
Pixie Autoencoder learns the prior, fitting the world model to the inference network's predictions. Since the world model makes structural assumptions, defining energy based only on semantic dependencies, we can see the world model as a "structural prior": the inference network is encouraged, via the first term in (8), to make predictions that can be modelled under these structural assumptions.
I have evaluated on two datasets, chosen for two reasons. Firstly, they allow a direct comparison with previous results (Emerson and Copestake, 2017b). Secondly, they require fine-grained semantic understanding, which starts to use the expressiveness of a functional model.
More open-ended tasks such as lexical substitution and question answering would require combining my model with additional components such as a semantic parser and a coreference resolver. Robust parsers exist which are compatible with my model (for example: Buys and Blunsom, 2017; Chen et al., 2018), but this would be a non-trivial extension, particularly for incorporating robust coreference resolution, which would ideally be done hand-in-hand with semantic analysis. Incorporating fine-grained semantics into such tasks is an exciting research direction, but beyond the scope of the current paper.
When reporting results, significance tests follow Dror et al. (2018).
I trained the model on WikiWoods (Flickinger et al., 2010; Solberg, 2012), which provides DMRS graphs (Copestake et al., 2005; Copestake, 2009) for  $55\mathrm{m}$  sentences (900m tokens) from the English Wikipedia (July 2008). It was parsed with the English Resource Grammar (ERG) (Flickinger, 2000, 2011) and PET parser (Callmeier, 2001; Toutanova et al., 2005), with parse ranking trained on We-Science (Ytrestol et al., 2009). It is updated with each ERG release; I used the 1212 version. I preprocessed the data following Emerson and Copestake (2016), giving  $31\mathrm{m}$  graphs.
I implemented the model using DyNet (Neubig et al., 2017) and Pydmrs (Copestake et al., 2016).<sup>6</sup> I initialised the generative model following Emerson and Copestake (2017b) using sparse PPMI vectors (QasemiZadeh and Kallmeyer, 2016). I first trained the encoder on the initial generative model, then trained both together. I used L2 regularisation and the Adam optimiser (Kingma and Ba, 2015), with separate L2 weights and learning rates for the world model, lexical model, and encoder. I tuned hyperparameters on the RELPRON dev set (see §4.3), and averaged over 5 random seeds.
4.2 BERT Baseline
BERT (Devlin et al., 2019) is a large pre-trained language model with a Transformer architecture (Vaswani et al., 2017), trained on 3.3b tokens from the English Wikipedia and BookCorpus (Zhu et al., 2015). It produces high-quality contextualised embeddings, but its architecture is not motivated by linguistic theory. I used the version in the Transformers library (Wolf et al., 2019). To my knowledge, large language models have not previously been evaluated on these datasets.
4.3 RELPRON
The RELPRON dataset (Rimell et al., 2016) consists of terms (such as telescope), paired with up to 10 properties (such as device that astronomer use). The task is to find the correct properties for each term. There is large gap between the state of the art (around  $50\%$ ) and the human ceiling (near  $100\%$ ).
The dev set contains 65 terms and 518 properties; the test set, 73 terms and 569 properties. The dataset is too small to train on, but hyperparameters can be tuned on the dev set. The dev and test terms are disjoint, to avoid high scores from overtuning.
$^{6}$ https://gitlab.com/guyemerson/pixie
Model	Dev	Test
Previous work	Vector addition (Rimell et al., 2016)	.496	.472
Simplified Practical Lexical Function (Rimell et al., 2016)	.496	.497
Vector addition (Czarnowska et al., 2019)	.485	.475
Dependency vector addition (Czarnowska et al., 2019)	.497	.439
Semantic functions (Emerson and Copestake, 2017b)	.20	.16
Sem-func & vector ensemble (Emerson and Copestake, 2017b)	.53	.49
Baselines	Vector addition	.488	.474
BERT (masked prediction)	.206	.186
BERT (contextual prediction)	.093	.134
BERT (masked prediction) & vector addition ensemble	.498	.479
Proposed approach	Pixie Autoencoder	.261	.189
Pixie Autoencoder & vector addition ensemble	.532	.489
Table 1: Mean Average Precision (MAP) on RELPRON development and test sets.
Previous work has shown that vector addition performs well on this task (Rimell et al., 2016; Czarnowska et al., 2019). I have trained a Skipgram model (Mikolov et al., 2013) using the Gensim library (Rehurek and Sojka, 2010), tuning weighted addition on the dev set.
For the Pixie Autoencoder, we can view the task as logical inference, finding the probability of truth of a term given an observed property. This follows Fig. 5, applying the term  $a$  to either  $X$  or  $Z$ , according to whether the property has a subject or object relative clause.
BERT does not have a logical structure, so there are multiple ways we could apply it. I explored many options, to make it as competitive as possible. Following Petroni et al. (2019), we can rephrase each property as a cloze sentence (such as a device that an astronomer uses is a [MASK].). However, RELPRON consists of pseudo-logical forms, which must be converted into plain text query strings. For each property, there are many possible cloze sentences, which yield different predictions. Choices include: grammatical number, articles, relative pronoun, passivisation, and position of the mask. I used the Pattern library (Smedt and Daelemans, 2012) to inflect words for number.
Results are given in Table 1. The best performing BERT method uses singular nouns with  $a/an$ , despite sometimes being ungrammatical. My most careful approach involves manually choosing articles (e.g. a device, the sky, water) and number (e.g. plural people) and trying three articles for the masked term ( $a, an$ , or no article, taking the highest probability from the three), but this actually lowers dev set performance to .192. Using plurals lowers performance to .089. Surprisingly, using
BERT large (instead of BERT base) lowers performance to .165. As an alternative to cloze sentences, BERT can be used to predict the term from a contextualised embedding. This performs worse (see Table 1), but the best type of query string is similar.
The Pixie Autoencoder outperforms previous work using semantic functions, but is still outperformed by vector addition. Combining it with vector addition in a weighted ensemble lets us test whether they have learnt different kinds of information. The ensemble significantly outperforms vector addition on the test set  $(p < 0.01$  for a permutation test), while the BERT ensemble does not  $(p > 0.2)$ . However, it performs no better than the ensemble in previous work. This suggests that, while the encoder has enabled the model to learn more information, the additional information is already present in the vector space model.
RELPRON also includes a number of confounders, properties that are challenging due to lexical overlap. For example, an activity that soil supports is farming, not soil. There are 27 confounders in the test set, and my vector addition model places all of them in the top 4 ranks for the confounding term. In contrast, the Pixie Autoencoder and BERT do not fall for the confounders, with a mean rank of 171 and 266, respectively.
Nonetheless, vector addition remains hard to beat. As vector space models are known to be good at topical relatedness (e.g. learning that astronomer and telescope are related, without necessarily learning how they are related), a tentative conclusion is that relatedness is missing from the contextualised models (Pixie Autoencoder and BERT). Finding a principled way to integrate a notion of "topic" would be an interesting task for future work.
Model	Separate	Averaged
Previous work	Vector addition (Milajevs et al., 2014)	-	.348
Categorical, copy object (Milajevs et al., 2014)	-	.456
Categorical, regression (Polajnar et al., 2015)	.33	-
Categorical, low-rank decomposition (Fried et al., 2015)	.34	-
Tensor factorisation (Van de Cruys et al., 2013)	.37	-
Neural categorical (Hashimoto et al., 2014)	.41	.50
Semantic functions (Emerson and Copestake, 2017b)	.25	-
Sem-func & vector ensemble (Emerson and Copestake, 2017b)	.32	-
Baselines	BERT (contextual similarity)	.337	.446
BERT (contextual prediction)	.233	.317
Proposed approach	Pixie Autoencoder (logical inference in both directions)	.306	.374
Pixie Autoencoder (logical inference in one direction)	.406	.504
Table 2: Spearman rank correlation on the GS2011 dataset, using separate or averaged annotator scores.
4.4 GS2011
The GS2011 dataset evaluates similarity in context (Grefenstette and Sadrzadeh, 2011). It comprises pairs of verbs combined with the same subject and object (for example, map show location and map express location), annotated with similarity judgements. There are 199 distinct pairs, and 2500 judgements (from multiple annotators).
Care must be taken when considering previous work, for two reasons. Firstly, there is no development set. Tuning hyperparameters directly on this dataset will lead to artificially high scores, so previous work cannot always be taken at face value. For example, Hashimoto et al. (2014) report results for 10 settings. I nonetheless show the best result in Table 2. My model is tuned on RELPRON ( $\S 4.3$ ).
Secondly, there are two ways to calculate correlation with human judgements: averaging for each distinct pair, or keeping each judgement separate. Both methods have been used in previous work, and only Hashimoto et al. (2014) report both.
For the Pixie Autoencoder, we can view the task as logical inference, following Fig. 5. However, Van de Cruys et al. (2013) point out that the second verb in each pair is often nonsensical when combined with the two arguments (e.g. system visit criterion), and so they argue that only the first verb should be contextualised, and then compared with the second verb. This suggests we should apply logical inference only in one direction: we should find the probability of truth of the second verb, given the first verb and its arguments. As shown in Table 2, this gives better results than applying logical inference in both directions and averaging the probabilities. Logical inference in both directions allows a direct comparison with Emerson and
Copestake (2017b), showing the Pixie Autoencoder performs better. Logical inference in one direction yields state-of-the-art results on par with the best results of Hashimoto et al. (2014).
There are multiple ways to apply BERT, as in §4.3. One option is to calculate cosine similarity of contextualised embeddings (averaging if tokenised into word-parts). However, each subject-verb-object triple must be converted to plain text. Without a dev set, it is reassuring that conclusions from RELPRON carry over: it is best to use singular nouns with  $a/an$  (even if ungrammatical) and it is best to use BERT base. Manually choosing articles and number lowers performance to .320 (separate), plural nouns to .175, and BERT large to .226. Instead of using cosine similarity, we can predict the other verb from the contextualised embedding, but this performs worse. The Pixie Autoencoder outperforms BERT, significantly for separate scores  $(p < 0.01$  for a bootstrap test), but only suggestively for averaged scores  $(p = 0.18)$ .
I have presented the Pixie Autoencoder, a novel encoder architecture and training algorithm for Functional Distributional Semantics, improving on previous results in this framework. For GS2011, the Pixie Autoencoder achieves state-of-the-art results. For RELPRON, it learns information not captured by a vector space model. For both datasets, it outperforms BERT, despite being a shallower model with fewer parameters, trained on less data. This points to the usefulness of building semantic structure into the model. It is also easy to apply to these datasets (with no need to tune query strings), as it has a clear logical interpretation.
Keith Allan. 2001. Natural language semantics. Blackwell.
Marco Baroni, Raffaela Bernardi, and Roberto Zamparelli. 2014. Frege in space: A program of compositional distributional semantics. Linguistic Issues in Language Technology (LiLT), 9.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3(Jan):993-1022.
Zied Bouraoui, Shoaib Jameel, and Steven Schockaert. 2017. Inductive reasoning about ontologies using conceptual spaces. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. Association for the Advancement of Artificial Intelligence.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), pages 10-21.
Jan Buys and Phil Blunsom. 2017. Robust incremental neural semantic graph parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), Long Papers, pages 1215-1226.
Ulrich Callmeier. 2001. Efficient parsing with large-scale unification grammars. Master's thesis, Saarland University.
Ronnie Cann. 1993. Formal semantics: an introduction. Cambridge University Press.
Yufei Chen, Weiwei Sun, and Xiaojun Wan. 2018. Accurate SHRG-based semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), Long Papers, pages 408-418.
Stephen Clark. 2015. Vector space models of lexical meaning. In Shalom Lappin and Chris Fox, editors, The Handbook of Contemporary Semantic Theory, 2nd edition, chapter 16, pages 493-522. Wiley.
Bob Coecke, Mehrnoosh Sadrzadeh, and Stephen Clark. 2010. Mathematical foundations for a compositional distributional model of meaning. Linguistic Analysis, 36, A Festschrift for Joachim Lambek:345-384.
Robin Cooper. 2005. *Austinian truth, attitudes and type theory*. Research on Language and Computation, 3(2-3):333-362.
Robin Cooper, Simon Dobnik, Staffan Larsson, and Shalom Lappin. 2015. Probabilistic type theory and natural language semantics. Linguistic Issues in Language Technology (LiLT), 10.
Ann Copestake. 2009. Slacker semantics: Why superficiality, dependency and avoidance of commitment can be the right way to go. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 1-9.
Ann Copestake, Guy Emerson, Michael Wayne Goodman, Matic Horvat, Alexander Kuhnle, and Ewa Muszyńska. 2016. Resources for building applications with Dependency Minimal Recursion Semantics. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC), pages 1240-1247. European Language Resources Association (ELRA).
Ann Copestake, Dan Flickinger, Carl Pollard, and Ivan A. Sag. 2005. Minimal Recursion Semantics: An introduction. Research on Language and Computation, 3(2-3):281-332.
Tim Van de Cruys, Thierry Poibeaun, and Anna Korhonen. 2013. A tensor-based factorization model of semantic compositionality. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies, pages 1142-1151.
Paula Czarnowska, Guy Emerson, and Ann Cope-stake. 2019. Words are vectors, dependencies are matrices: Learning word embeddings from dependency graphs. In Proceedings of the 13th International Conference on Computational Semantics (IWCS), Long Papers, pages 91-102. Association for Computational Linguistics.
Donald Davidson. 1967. The logical form of action sentences. In Nicholas Rescher, editor, *The Logic of Decision and Action*, chapter 3, pages 81-95. University of Pittsburgh Press. Reprinted in: Davidson (1980/2001), *Essays on Actions and Events*, Oxford University Press.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1 (Long Papers), pages 1383-1392.
Michael Dummett. 1976. What is a theory of meaning? (II). In Gareth Evans and John McDowell, editors, Truth and Meaning: Essays in Semantics, chapter 4, pages 67-137. Clarendon Press (Oxford). Reprinted in: Dummett (1993), Seas of Language, chapter 2, pages 34-93.
Michael Dummett. 1978. What do I know when I know a language? Presented at the Centenary Celebrations of Stockholm University. Reprinted in: Dummett (1993), *Seas of Language*, chapter 3, pages 94–105.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. 2015. Convolutional networks on graphs for learning molecular fingerprints. In Advances in Neural Information Processing Systems (NeurIPS), pages 2224-2232.
Guy Emerson. 2018. *Functional Distributional Semantics: Learning Linguistically Informed Representations from a Precisely Annotated Corpus*. Ph.D. thesis, University of Cambridge.
Guy Emerson. 2020a. What are the goals of distributional semantics? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL).
Guy Emerson. 2020b. Linguists who use probabilistic models love them: Quantification in Functional Distributional Semantics. In Proceedings of Probability and Meaning (PaM2020). Association for Computational Linguistics.
Guy Emerson and Ann Copestake. 2016. Functional Distributional Semantics. In Proceedings of the 1st Workshop on Representation Learning for NLP (RepL4NLP), pages 40-52. Association for Computational Linguistics.
Guy Emerson and Ann Copestake. 2017a. Variational inference for logical inference. In Proceedings of the Conference on Logic and Machine Learning in Natural Language (LaML), pages 53-62. Centre for Linguistic Theory and Studies in Probability (CLASP).
Guy Emerson and Ann Copestake. 2017b. Semantic composition via probabilistic model theory. In Proceedings of the 12th International Conference on Computational Semantics (IWCS), pages 62-77. Association for Computational Linguistics.
Katrin Erk. 2009a. Supporting inferences in semantic space: representing words as regions. In Proceedings of the 8th International Conference on Computational Semantics (IWCS), pages 104-115. Association for Computational Linguistics.
Katrin Erk. 2009b. Representing words as regions in vector space. In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL), pages 57-65. Association for Computational Linguistics.
Katrin Erk. 2012. Vector space models of word meaning and phrase meaning: A survey. Language and Linguistics Compass, 6(10):635-653.