Complex Network Analysis of Distributional Semantic Models

Complex Network Analysis of Distributional Semantic Models Akira Utsumi (utsumi@inf.uec.ac.jp) Department of Informatics, The University of Electro-Communications 1-5-1, Chofugaoka, Chofushi, Tokyo 182-8585, Japan Abstract A number of studies on network analysis have found the small- world and scale-free properties in the network of free word association, which reflects human semantic knowledge. Nev- ertheless, there have been very few attempts to apply network analysis to distributional semantic models (DSMs), despite the fact that DSMs have been extensively studied as a model of human semantic knowledge. In this paper, therefore, we an- alyze the small-world and scale-free properties of DSM net- works. We demonstrate that DSM networks exhibit the same properties as the word association network. Especially, we show that DSM networks have the distribution of the number of connections that follows the truncated power law, which is also observed in the association network. This result indicates that DSMs provide a plausible model of semantic knowledge. Furthermore, we propose a modified version of Steyvers and Tenenbaum’s (2005) growing network model, which involves the processes of semantic differentiation and experiential cor- relation. This model can better explain different distributions generated by various DSM implementations. Keywords: Semantic network; Distributional semantic model; Truncated power-law distributions; Network models Introduction Recently, network analysis or network science has attracted considerable attention in cognitive science (Baronchelli, Ferrer-i-Cancho, Pastor-Satorras, Chater, & Christiansen, 2013). The network properties revealed through the anal- ysis of complex cognitive phenomena tell us about the be- havior of the underlying cognitive processes, and even sim- ple network models can provide valuable insights into the cognitive mechanisms. In particular, a large number of net- work studies have investigated language-related phenomena (Borge-Holthoefer & Arenas, 2010), among which the most studied one is free word association (De Deyne & Storms, 2008; Nelson, McEvoy, & Schreiber, 2004). This is because free word association reflects our lexical knowledge acquired through world experience, and thus reveals the structure of human semantic memory or mental lexicon. Network studies on word association have demonstrated the small-world and scale-free properties of semantic network (De Deyne & Storms, 2008; Morais, Olsson, & Schooler, 2013; Steyvers & Tenenbaum, 2005). For an association net- work where each word is represented by a node and an asso- ciation relation between two words is represented by an edge joining the corresponding nodes, the small-world property in- dicates that any two word nodes are connected by traversing only a few edges, although the network is highly structured locally. The scale-free property indicates that most word nodes are poorly connected, while a relatively small number of words are highly connected; as a result, the distribution of the number of connections for each node follows a power law. All the existing studies agree on the small-world property of the association network, but some studies (Morais et al., 2013) suggest that the association network is not completely scale-free; rather the network is characterized by a power law truncated by an exponential cutoff, where the most connected words have a smaller connection than would be expected in a purely power-law distributed network. These network properties are expected to reveal the cog- nitive mechanism underlying the semantic structure of lan- guage (Borge-Holthoefer & Arenas, 2010). For example, the small-world structure sheds light on an efficient search process in semantic memory. Investigating various network models that generate scale-free networks provides valuable insight about psychological processes involved in lexical de- velopment. In contrast to the growing network-analytic interest in word association, a distributional semantic model (hence- forth, DSM) has rarely been investigated in network analy- sis, despite the fact that DSMs have been extensively studied as a valid model of semantic memory (Landauer, McNamara, Dennis, & Kintsch, 2007). In a DSM, the lexical meaning of a word is represented by a high-dimensional vector, and the degree of semantic relatedness between any two words can be easily computed from their vectors. Word vectors are constructed from large bodies of text (i.e., corpus) by ob- serving distributional statistics of word occurrence. Steyvers and Tenenbaum (2005) tested whether latent semantic anal- ysis (LSA), one of the most popular versions of DSMs, ex- hibits the same network structure as word association, and found that LSA networks were small-world, but not scale- free. They concluded from this result that LSA is limited as a model of human semantic knowledge. However, their finding does not imply that DSMs in general fail to model semantic memory, because a variety of methods for constructing se- mantic spaces other than LSA are devised in the DSM frame- work (e.g., Turney & Pantel, 2010). Furthermore, as Morais et al. (2013) pointed out, their analysis of the scale-free prop- erty was quite subjective in that their claim of power-law be- havior was derived solely from the observation of the behav- ior of distribution without any statistical tests. In this paper, we analyze the small-world and scale-free properties of semantic networks constructed from various DSMs in a more systematic way, and examine the condi- tions under which DSM networks have the same properties as the association network. Through this network analysis, we test whether a DSM can provide a psychologically plau- sible model of semantic memory. In addition, we discuss a cognitive mechanism for lexical development by proposing a network model that simulates the behavior of DSM networks. Complex Network Analysis We first define some terminology from graph theory used in this paper. A network (or graph) G is defined by a pair (V, E) comprising a set V of nodes or vertices and a set E of edges that connect a pair of nodes. The number of nodes |V | is denoted by n, and the number of edges |E| is denoted by m. An edge is directed or undirected, and a graph containing only directed (undirected) edges is said to be directed (undirected). Two nodes are neighbors if they are connected by an edge. The degree k i of a node v i is the number of edges that connect to it (i.e., the number of neighbors). The average degree over all nodes is denoted by k. In a directed network, the degree of a node v i has two types, namely in-degree k i in and out- degree k i out , which respectively refer to the numbers of edges incoming to or outgoing from v i . A path is a sequence of edges that connects one node to