SlowGuess/ACL_2020_OCR huggingface dataset:
Thematic-Fit Evaluation (2 Variables: SV and VO). Similarly to the 3-group setup, we also evaluate the plausibility of  $SV$  and  $VO$  pairs separately in the 2-group setup. The selectional preference evaluation (Sayeed et al., 2016), also referred to as thematic-fit, quantifies the extent to which a noun fulfils the selectional preference of a verb given a role (i.e., agent:S, or patient:O) (McRae et al., 1997). We evaluate our 2-group function-specific
Data set	Train	Test
SVO+iO	187K	15K
SVO	22M	214K
Vocab size	Freq.
S	22K	people,one,company,student
V	5K	have,take,include,provide
O	15K	place,information,way,number
SV	69M	232K
Vocab size	Freq.
S	45K	people,what,one,these
V	19K	be,have,say,take,go
VO	84M	240K
Vocab size	Freq.
V	9K	have,take,use,make,provide
O	32K	information,time,service
Table 2: Training data statistics.
Model	Accuracy
4 Variables
SVO+iO	0.950
3 Variables: SVO
Van de Cruys (2009)	0.874
Van de Cruys (2014)	0.889
Tilk et al. (2016)◇	0.937
Ours	0.943
2 Variables
Rooth et al. (1999)	0.720
Erk et al. (2010)	0.887
Van de Cruys (2014)	0.880
Ours: SV	0.960
Ours: VO	0.972
Table 3: Accuracy scores on the pseudo disambiguation task.  $\diamond$  indicates our reimplementation.
spaces on two standard benchmarks: 1) MST1444 (McRae et al., 1998) contains 1,444 word pairs where humans provided thematic fit ratings on a scale from 1 to 7 for each noun to score the plausibility of the noun taking the agent role, and also taking the patient role.3 2) PADO414 (Padó, 2007) is similar to MST1444, containing 414 pairs with human thematic fit ratings, where role-filling nouns were selected to reflect a wide distribution of scores for each verb. We compute plausibility by simply taking the cosine similarity between the verb vector (from the  $V$  space) and the noun vector from the appropriate function-specific space ( $S$  space for agents;  $O$  space for patients). We again report Spearman's  $\rho$  correlation scores.
3Using an example from Sayeed et al. (2016), the human participants were asked "how common is it for a {snake, monster, baby, cat} to frighten someone/something" (agent role) as opposed to "how common is it for a {snake, monster, baby, cat} to be frightened by someone/something" (patient role).
For instance, the phrases 'people run company' and 'people operate company' have a high similarity score of 6.53, whereas 'river meet sea' and 'river satisfy sea' have been given a low score of 1.84.
Training Data. We parse the ukWaC corpus (Baroni et al., 2009) and the British National Corpus (BNC) (Leech, 1992) using the Stanford Parser with Universal Dependencies v1.4 (Chen and Manning, 2014; Nivre et al., 2016) and extract co-occurring subjects, verbs and objects. All words are lowercased and lemmatised, and tuples containing non-alphanumeric characters are excluded. We also remove tuples with (highly frequent) pronouns as subjects, and filter out training examples containing words with frequency lower than 50. After preprocessing, the final training corpus comprises 22M SVO triplets in total. Table 2 additionally shows training data statistics when training in the 2-group setup (SV and VO) and in the 4-group setup (when adding indirect objects: SVO+iO). We report the number of examples in training and test sets, as well as vocabulary sizes and most frequent words across different categories.
Hyperparameters. We train with batch size 128, and use Adam for optimisation (Kingma and Ba, 2015) with a learning rate 0.001. All gradients are clipped to a maximum norm of 5.0. All models were trained with the same fixed random seed. We train 25-dimensional vectors for all setups (2/3/4 groups), and we additionally train 100-dimensional vectors for the 3-group (SVO) setup.
Pseudo-Disambiguation. Accuracy scores on the pseudo-disambiguation task in the 2/3/4-group setups are summarised in Table 3. $^{4}$  We find consistently high pseudo-disambiguation scores  $(>0.94)$  across all setups. In a more detailed analysis, we find especially the prediction accuracy of verbs to be high: we report accuracy of  $96.9\%$  for the 3-group SVO model. The vocabulary size for verbs is typically lowest (see Table 2), which presumably makes predictions into this direction easier. In summary, as mentioned in Section 5, this initial evaluation already suggests that our model is able to capture associations between interrelated groups which are instrumental to modeling SVO structures and composing event representations.
Event Similarity. We now test correlations of SVO-based event representations composed from a
Composition	Reference	Formula
Verb only	Milajevs et al. (2014)	V̄
Addition	Mitchell and Lapata (2008)	S + V̄ + O
Copy Object	Kartsaklis et al. (2012)	S ⊙ (V × O)
Concat	Edelstein and Reichart (2016)	[S, V̄, O]
Concat Addition	Edelstein and Reichart (2016)	[S, V̄] + [V, O]
Network	Ours	SVT + VOT + SOT
Table 4: Composition functions used to obtain event vectors from function-specific vector spaces. +: addition, ⊙: element-wise multiplication, ×: dot product. [·, ·]: concatenation.
Model	Reference	Spearman's ρ
GS199	KS108
Copy Object W2V	Milajevs et al. (2014)	0.46	0.66
Addition KS14	Milajevs et al. (2014)	0.28	0.73
Tilk et al. (2016)	0.34	-
Weber et al. (2018)	-	0.71
Ours: SVO d100
Verb only	Ours	0.34	0.63
Addition	Ours	0.27	0.76
Concat	Ours	0.26	0.75
Concat Addition	Ours	0.32	0.77
Copy Object	Ours	0.40	0.52
Network	Ours	0.53	-
Table 5: Results on the event similarity task. Best baseline score is underlined, and the best overall result is provided in bold.
function-specific vector space (see Table 4) to human scores in the event similarity task. A summary of the main results is provided in Table 5. We also report best baseline scores from prior work. The main finding is that our model based on function-specific word vectors outperforms previous state-of-the-art scores on both datasets. It is crucial to note that different modeling approaches and configurations from prior work held previous peak scores on the two evaluation sets. Interestingly, by relying only on the representations from the  $V$  subspace (i.e., by completely discarding the knowledge stored in  $S$  and  $O$  vectors), we can already obtain reasonable correlation scores. This is an indicator that the verb vectors indeed stores some selectional preference information as designed, i.e., the information is successfully encoded into the verb vectors themselves.
Thematic-Fit Evaluation. Correlation scores on two thematic-fit evaluation data sets are summarised in Table 6. We also report results with
Note the two tasks are inherently different. KS108 requires similarity between plausible triplets. Using the network score directly (which is a scalar, see Table 4) is not suitable for KS108 as all KS108 triplets are plausible and scored highly. This is reflected in the results in Table 5.
4We also provide baseline scores taken from prior work, but the reader should be aware that the scores may not be directly comparable due to the dependence of this evaluation on factors such as vocabulary size and sampling of corrupted examples (Chambers and Jurafsky, 2010; Sayeed et al., 2016).
representative baseline models for the task: 1) a TypeDM-based model (Baroni and Lenci, 2010), further improved by Greenberg et al. (2015a,b) (G15), and 2) current state-of-the-art tensor-based neural model by Tilk et al. (2016) (TK16). We find that vectors taken from the model trained in the joint 3-group SVO setup perform on a par with state-of-the-art models also in the 2-group evaluation on SV and VO subsets. Vectors trained explicitly in the 2-group setup using three times more data lead to substantial improvements on PADO414. As a general finding, our function-specific approach leads to peak performance on both data sets. The results are similar with 25-dim SVO vectors.
Our model is also more light-weight than the baselines: we do not require a full (tensor-based) neural model, but simply function-specific word vectors to reason over thematic fit. To further verify the importance of joint multidirectional training, we have also compared our function-specific vectors against standard single-space word vectors (Mikolov et al., 2013b). The results indicate the superiority of function-specific spaces: respective correlation scores on MST1444 and PADO414 are 0.28 and 0.41 (vs 0.34 and 0.58 with our model). It is interesting to note that we obtain state-of-the-art scores calculating cosine similarity of vectors taken from two groups found in the joint space. This finding verifies that the model does indeed learn a joint space where co-occurring words from different groups lie close to each other.
Qualitative Analysis. We retrieve nearest neighbours from the function-specific  $(S, V, O)$  space, shown in Figure 1. We find that the nearest neighbours indeed reflect the relations required to model the SVO structure. For instance, the closest subjects/agents to the verb eat are cat and dog. The closest objects to need are three plausible nouns: help, support, and assistance. As the model has information about group membership, we can also filter and compare nearest neighbours in single-group subspaces. For example, we find subjects similar to the subject memory are dream and feeling, and objects similar to beer are ale and pint.
Model Variants. We also conduct an ablation study that compares different model variants. The variants are constructed by varying 1) the training regime: asynchronous (async) vs synchronous (sync), and 2) the type of parameter sharing: training on separate parameters for each sub-network
Setup	Baselines	Ours
Dataset	Eval	G15	TK16	SVO (d=100)	SV-VO (d=25)
MST1444	SV	0.36	-	0.37	0.31
VO	0.34	-	0.35	0.35
full	0.33	0.38	0.36	0.34
PADO414	SV	0.54	-	0.38	0.55
VO	0.53	-	0.54	0.61
full	0.53	0.52	0.45	0.58
Table 6: Results on the 2-variable thematic-fit evaluation. Spearman's  $\rho$  correlation.
async	sync
sep	shared	sep	shared
3 Variables
KS108 Verb only	0.56	0.48	0.58	0.60
KS108 Addition	0.51	0.66	0.73	0.78
GS199 Verb only	0.24	0.26	0.26	0.34
GS199 Network	0.10	0.40	0.28	0.52
2 Variables
MST1444	0.17	0.10	0.30	0.39
PADO414	0.41	0.21	0.44	0.44
Table 7: Evaluation of different model variants, by training regime and parameter sharing.
$(sep)^6$  or training on shared variables (shared). In the asynchronous setup we update the shared parameters per sub-network directly based on their own loss, instead of relying on the joint synchronous loss as in Section 3.
Table 7 shows the results with the model variants, demonstrating that both aspects (i.e., shared parameters and synchronous training) are important to reach improved overall performance. We reach the peak scores on all evaluation sets using the sync+shared variant. We suspect that asynchronous training deteriorates performance because each sub-network overwrites the updates of other subnetworks as their training is not tied through a joint loss function. On the other hand, the synchronous training regime guides the model towards making updates that can benefit all sub-networks.
We presented a novel multidirectional neural framework for learning function-specific word representations, which can be easily composed into multi-word representations to reason over event similarity and thematic fit. We induced a joint vector space
<sup>6</sup>With separate parameters we merge vectors from "duplicate" vector spaces by non-weighted averaging.
in which several groups of words (e.g., S, V, and O words forming the SVO structures) are represented while taking into account the mutual associations between the groups. We found that resulting function-specific vectors yield state-of-the-art results on established benchmarks for the tasks of estimating event similarity and evaluating thematic fit, previously held by task-specific methods.
In future work we will investigate more sophisticated neural (sub-)networks within the proposed framework. We will also apply the idea of function-specific training to other interrelated linguistic phenomena and other languages, probe the usefulness of function-specific vectors in other language tasks, and explore how to integrate the methodology with sequential models. The pre-trained word vectors used in this work are available online at:
https://github.com/cambridgeltl/fs-wrep.
This work is supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909) awarded to Anna Korhonen.
Andrew Anderson, Douwe Kiela, Stephen Clark, and Massimo Poesio. 2017. Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns. Transactions of the ACL, 5:17-30.
Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: a collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209-226.
Marco Baroni and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673-721.
Shane Bergsma, Dekang Lin, and Randy Goebel. 2008. Discriminative learning of selectional preference from unlabeled text. In Proceedings of EMNLP, pages 59-68.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the ACL, 5:135-146.
Jose Camacho-Collados, Luis Espinosa Anke, and Steven Schockaert. 2019. Relational word embeddings. In Proceedings of ACL, pages 3286-3296.
Alfonso Caramazza and Argye E. Hillis. 1991. Lexical organization of nouns and verbs in the brain. Nature, 349(6312):788-790.
Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In Proceedings of ACL, pages 602-610.
Nathanael Chambers and Dan Jurafsky. 2010. Improving the use of pseudo-words for evaluating selectional preferences. In Proceedings of ACL, pages 445-453.
Danqi Chen and Christopher D. Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of EMNLP, pages 740-750.
Bob Coecke, Mehrnoosh Sadrzadeh, and Stephen Clark. 2010. Mathematical foundations for a compositional distributional model of meaning. Linguistic Analysis, 36(1-4):345-384.
Ronan Collobert, Jason Weston, Lon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493-2537.
Tim Van de Cruys. 2009. A non-negative tensor factorization model for selectional preference induction. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 83-90.
Tim Van de Cruys. 2014. A neural network approach to selectional preference acquisition. In Proceedings of EMNLP, pages 26-35.
Antonio R. Damasio and Daniel Tranel. 1993. Nouns and verbs are retrieved with differently distributed neural systems. Proceedings of the National Academy of Sciences of the United States of America, 90(11):4957-60.
R. Rhys Davies, Glenda M. Halliday, John H. Xuereb, Jillian J. Kril, and John R. Hodges. 2009. The neural basis of semantic memory: Evidence from semantic dementia. Neurobiology of Aging, 30(12):2043-2052.
Lilach Edelstein and Roi Reichart. 2016. A factorized model for transitive verbs in compositional distributional semantics. CoRR, abs/1609.07756.
Guy Emerson and Ann A. Copestake. 2016. Functional distributional semantics. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 40-52.
Katrin Erk, Sebastian Padó, and Ulrike Padó. 2010. A flexible, corpus-driven model of regular and inverse selectional preferences. Computational Linguistics, 36(4):723-763.
Manaal Faruqui. 2016. Diverse Context for Learning Word Representations. Ph.D. thesis, Carnegie Mellon University.
Murray Gell-Mann and Merritt Ruhlen. 2011. The origin and evolution of word order. Proceedings of the National Academy of Sciences, 108(42):17290-17295.
Clayton Greenberg, Vera Demberg, and Asad Sayeed. 2015a. Verb polysemy and frequency effects in the automatic fit modeling. In Proceedings of the 6th Workshop on Cognitive Modeling and Computational Linguistics, pages 48-57.
Clayton Greenberg, Asad Sayeed, and Vera Demberg. 2015b. Improving unsupervised vector-space thematic fit evaluation via role-filler prototype clustering. In Proceedings of NAACL-HLT, pages 21-31.
Edward Grefenstette and Mehrnoosh Sadrzadeh. 2011a. Experimental support for a categorical compositional distributional model of meaning. In Proceedings of EMNLP, pages 1394-1404.
Edward Grefenstette and Mehrnoosh Sadrzadeh. 2011b. Experimenting with transitive verbs in a DisCoCat. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pages 62-66.
Zellig S. Harris. 1954. Distributional Structure. Word, 10(2-3):146-162.
Kazuma Hashimoto and Yoshimasa Tsuruoka. 2016. Adaptive joint learning of compositional and non-compositional phrase embeddings. In Proceedings of ACL, pages 205-215.
Wendy A. de Heer, Alexander G. Huth, Thomas L. Griffiths, Jack L. Gallant, and Frédéric E. Theunissen. 2017. The hierarchical cortical organization of human speech processing. Journal of Neuroscience, 37(27):6539-6557.
Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665-695.
Alexander G. Huth, Wendy A. de Heer, Thomas L. Griffiths, Frédéric E. Theunissen, and Jack L. Gallant. 2016. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):453-458.
Alexander G. Huth, Shinji Nishimoto, An T. Vu, and Jack L. Gallant. 2012. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. *Neuron*, 76(6):1210-1224.
Dimitri Kartsaklis and Mehrnoosh Sadrzadeh. 2014. A study of entanglement in a categorical framework of natural language. In Proceedings of QPL, pages 249-261.
Dimitri Kartsaklis, Mehrnoosh Sadrzadeh, and Stephen Pulman. 2012. A unified sentence space for categorical distributional-compositional semantics: Theory
and experiments. In Proceedings of COLING, pages 549-558.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR (Conference Track).
Geoffrey Neil Leech. 1992. 100 million words of English: The British National Corpus (BNC).
Omer Levy and Yoav Goldberg. 2014a. Dependency-based word embeddings. In Proceedings of ACL, pages 302-308.
Omer Levy and Yoav Goldberg. 2014b. Neural word embedding as implicit matrix factorization. In Proceedings of NIPS, pages 2177-2185.
Laurens van der Maaten and Geoffrey E. Hinton. 2012. Visualizing non-metric similarities in multiple maps. Machine Learning, 87(1):33-55.
Rosaleen A. McCarthy and E. K. Warrington. 1988. Evidence for modality-specific meaning systems in the brain. Nature, 334(6181):428-430.
Ken McRae, Todd Ferretti, and Liane Amyote. 1997. Thematic roles as verb-specific concepts. Language and Cognitive Processes, 12(2):137-176.
Ken McRae, Michael J. Spivey-Knowlton, and Michael K. Tanenhaus. 1998. Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language, 38(3):283-312.
Oren Melamud, David McClosky, Siddharth Patwardhan, and Mohit Bansal. 2016. The role of context types and dimensionality in learning word embeddings. In Proceedings of NAACL-HLT, pages 1030-1040.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of ICLR (Workshop Papers).
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS, pages 3111-3119.
Dmitrijs Milajevs, Dimitri Kartsaklis, Mehrnoosh Sadrzadeh, and Matthew Purver. 2014. Evaluating neural word representations in tensor-based compositional settings. In Proceedings of EMNLP, pages 708-719.
Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. Proceedings of ACL, pages 236-244.
Tom M. Mitchell, Svetlana V. Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L. Malave, Robert A. Mason, and Marcel Adam Just. 2008. Predicting human brain activity associated with the meanings of nouns. Science, 320(5880):1191-1195.
2881
Ashutosh Modi. 2016. Event embeddings for semantic script modeling. In Proceedings of CoNLL, pages 75-83.
Nikola Mrkšić, Ivan Vulic, Diarmuid Řeaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017. Semantic specialisation of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the ACL, 5:309–324.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan T McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of LREC, pages 1659-1666.
Ulrike Padó. 2007. The integration of syntax and semantic plausibility in a wide-coverage model of human sentence processing.
Belen Pascual, Joseph C. Masdeu, Mark Hollenbeck, Nikos Makris, Ricardo Insausti, Song-Lin Ding, and Bradford C. Dickerson. 2015. Large-scale brain networks of the human left temporal pole: A functional connectivity MRI study. *Cerebral Cortex*, 25(3):680-702.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532-1543.
Karl Pichotta and Raymond J. Mooney. 2016. Learning statistical scripts with LSTM recurrent neural networks. In Proceedings of AAAI, pages 2800-2806.
Marek Rei, Daniela Gerz, and Ivan Vulic. 2018. Scoring lexical entailment with a supervised directional similarity network. In Proceedings of ACL, pages 638-643.
Grace E. Rice, Paul Hoffman, and Matthew A. Lambon Ralph. 2015. Graded specialization within and between the anterior temporal lobes. Annals of the New York Academy of Sciences, 1359(1):84-97.
M. Jane Riddoch, Glyn W. Humphreys, Max Coltheart, and Elaine Funnell. 1988. Semantic systems or system? Neuropsychological evidence re-examined. Cognitive Neuropsychology, 5(1):3-25.
Mats Rooth, Stefan Riezler, Detlef Prescher, Glenn Carroll, and Franz Beil. 1999. Inducing a semantically annotated lexicon via EM-based clustering. In Proceedings of ACL, pages 104-111.
Sebastian Ruder. 2017. An overview of multitask learning in deep neural networks. CoRR, abs/1706.05098.
Asad Sayeed, Clayton Greenberg, and Vera Demberg. 2016. Thematic fit evaluation: An aspect of selectional preferences. In Proceedings of the 1st Workshop on Evaluating Vector Space Representations for NLP, pages 99-105.
Hinrich Schütze. 1993. Word space. In Proceedings of NIPS, pages 895-902.
Roy Schwartz, Roi Reichart, and Ari Rappoport. 2015. Symmetric pattern based word embeddings for improved word similarity prediction. In Proceedings of CoNLL, pages 258-267.
Ottokar Tilk, Vera Demberg, Asad Sayeed, Dietrich Klakow, and Stefan Thater. 2016. Event participant modelling with neural networks. In Proceedings of EMNLP, pages 171-182.
Ivan Vulic and Nikola Mrkšić. 2018. Specialising word vectors for lexical entailment. Proceedings of NAACL-HLT.
Elizabeth K. Warrington. 1975. The Selective Impairment of Semantic Memory. Quarterly Journal of Experimental Psychology, 27(4):635-657.
Elizabeth K. Warrington and Rosaleen A. McCarthy. 1987. Categories of knowledge. *Brain*, 110(5):1273-1296.
Noah Weber, Niranjan Balasubramanian, and Nathanael Chambers. 2018. Event representations with tensor-based compositions. In Proceedings of AAAI, pages 4946-4953.