Why do embedding spaces look as they do?

1 INTRODUCTION . Vector representations ( Mikolov et al. , 2013a ; b ; Pennington et al. , 2014 ; Perozzi et al. , 2014 ) of feature spaces ( embeddings ) have proven amazingly effective at capturing the properties of discrete entities such as vocabulary words and the vertices of networks capturing human interactions , such as citation graphs or Wikipedia pages . Such embeddings are a foundational tool in machine learning , providing a natural data representation easily applied for classification and regression models . The power of embedding representations is a curious phenomenon . That embeddings function so effectively as feature representations implies there must exist substantial latent structure inherent in each domain to be effectively encoded . Language vocabularies and Wikipedia topics are representative human-generated structures that reflect how people organize their world , and what they find important . The structure of the resulting embedding spaces thus must reflect the human evolution of language formation and the cultural processes shaping our world . This paper addresses a big picture question : what does the observed structure of embeddings tell us about the processes by which people generate new knowledge or concepts ? Word and graph embeddings provide reduced-dimensional “ geometric ” representation of the meaning of words and conceptual entities . The widespread success of predictive models trained on top of embeddings suggest that they accurately capture the underlying semantics of each entity . The question we study in this paper concerns which knowledge-generation models best explain properties observed in realworld embeddings . The vocabulary of every language evolves through a series of processes : new words get created to identify new concepts or objects , while old words get repurposed with new senses and meanings . But how/where does this creation happen ? Word usage frequencies are governed by a power law distribution , namely Zipf ’ s law ( Zipf , 1936 ; 1949 ) . Do new words tend to be closely associated with high frequency words , perhaps adding shades of meaning to popular terms ? Do they tend to fill large empty regions of embedding space , or add to dense clusters of points ? If embeddings do indeed capture the meaning of words , then studying the shape of embedding spaces should inform on the processes underlying the evolution of language . A similar set of questions can be asked about the evolution of cultural , historical and scientific knowledge , as reflected by the contents of Wikipedia . Each entity page in Wikipedia links to other entities , creating a network well represented by its graph embedding . Do high-degree historical entities disproportionately associate with other high-degree entities ? Do new historical figures emerge in the context of clusters of earlier entities , or do they fill empty niches in the cultural landscape ? A variety of generative models can be proposed to build embedding-like spaces by incrementally inserting new points positioned through a specific algorithmic process . We seek to compare the properties of such generated spaces against those of observed embedding spaces , to gain insight into the processes underlying human knowledge generation activities . This is complicated by several factors : ( 1 ) we can not directly observe or reconstruct the historical processes that created the entities of observed data sets , ( 2 ) there is no direct correspondence between the labels of generated and observed points , and ( 3 ) the scale and range of real/synthetic embeddings over different models , algorithms , and datasets are not directly comparable . Thus assessments must be done through nonparametric statistics quantifying similarities of unlabeled point sets . Our contributions here include : • Generative processes for embedding spaces . We define and explore the space of several plausible processes for the evolution of knowledge embedding spaces , incorporating into our models the mechanisms of preferential attachment as well as the effects of attractive and/or repulsive forces . Certain processes generate networks to embed using standard graph representation learning methods , while others directly generate geometric point sets . Particularly novel is our class of preferential placement models . • Non-parametric properties of embedding spaces . Meaningful comparisons of the properties of observed and generated embedding spaces requires statistics which are robust to transformations as well as being independent of node labels . We have identified two novel and distinct properties of interest which can be rigorously measured on embedding spaces : – Frequency concentration – How the importance of entities in the local neighborhood of x is related to the importance of x itself ? This statistic captures the non-uniform spatial distribution of frequencies observed in an embedding . – Clustering velocity – How rapidly do clustering procedures advance towards full connectivity ? This statistic captures the predominance of cluster structure inherent in an embedding . As a check , we also compare these statistics to measures of spatial correlation popular in geographic analysis . Our tools allow to uncover novel regularities in the natural embedding spaces : elements of embeddings are not only characterized by power-law distributions of their frequencies/degrees , but they also exhibit positive assortative sorting in a spatial dimension and a power law manifestation in relational space . Moreover , we show that these observed properties of English language vocabularies and Wikipedia network embeddings consistently hold for a variety of distinct data sources and embedding algorithms , including fastText ( Bojanowski et al. , 2017 ) , GloVe ( Pennington et al. , 2014 ) , word2vec ( Mikolov et al. , 2013a ) , and BERT ( Devlin et al. , 2018 ) . Thus they are not mere algorithmic artifacts of data collection or analysis , but inherent properties of language and culturally-generated networks : phenomena in need of explanation through models of language/knowledge creation . • Evaluation of generative processes for word and graph embedding spaces . We compare embeddings generated by several different processes to real data-driven embeddings from several datasets by these criteria . Several models can easily be eliminated based on these observations , leaving few that remain plausible . We conclude that specific incremental insertion processes based on the spatial context-dependent preferential attachment mechanisms best model the observed phenomenon on language and network data . Paper outline . The rest of this paper is organized as follows . Section 2 defines and presents the collection of seven different models we consider as potential generative processes . Section 3 proposes our frequency concentration statistic and evaluates embeddings from both models and real-world ones . We present our clustering velocity statistics and model evaluations in Section 4 . Finally , we discuss the implications of this research and future work in Section 5 . Our code and datasets are accessible at https : //shorturl.at/dyST6 1.1 RELATED WORK . Explanations of embedding spaces . Levy & Goldberg ( 2014 ) find that word embedding model such as word2vec essentially factorizes a shifted pointwise mutual information matrix . To further understand the linear relationship of embeddings , Allen & Hospedales ( 2019 ) ; Allen et al . ( 2019 ) mathematically define the meaning of statement “ word vector wx is to word vector wy ” and provide a proof of the linearity among embeddings of word analogies . This linear property has also been explained in knowledge graph embeddings ( Allen et al. , 2021 ) . However , this line of work is more focusing on the linear property between embedding vectors instead of the processes of generation . Generative models of embeddings . There are many graph/network generative models ( see the survey Hamilton ( 2020 ) and references therein ) . Traditional processes include the Erdős–Rényi model ( Erdos et al. , 1960 ) , stochastic block model ( Holland et al. , 1983 ) , and preferential attachment models ( Simon ( 1955 ) , Barabási & Albert ( 1999 ) , also see Albert & Barabási ( 2002 ) for a review ) . Arora et al . ( 2016 ) study a generative model where each word vector captures its correlations with the discourse vector . Instead , our generative models employ simple heuristics to generate embedding vectors that exhibit empirically realistic properties . There are also works on generating models by using deep graph neural networks ( You et al. , 2018 ) . Xiao et al . ( 2016 ) propose a model based on Dirichlet Process to generate embeddings of knowledge graph . Li et al . ( 2016 ) propose a graphical model for generating embeddings of documents . Most of these models are heavily based on a probabilistic framework . Notation . For a set of n entities ( e.g . words or nodes ) V : = { 1 , 2 , . . . , n } , the set of embeddings associated with V is { w1 , w2 , . . . , wn } , where each location or whereabouts wi ∈ Rd . Throughout this paper , each entity i is also associated with a importance or prominence ( e.g . the frequency of a specific words , or the degree of a graph node ) and denoted as pi . Therefore , each entity i can be represented as pair of importance and location ⟨pi , wi⟩ . We collect them in an embedding matrix W ∈ Rn×d and an importance vector p ∈ Rn . 2 GENERATIVE MODELS FOR EMBEDDING SPACES . A generative model of an n-element embedding space starts with an initial entity embedding pair ⟨p0 , w0⟩ . At each time t = 1 , 2 , . . . , n , the model then repeatedly generates a new pair ⟨pt , wt⟩ . During this generating process , it updates p and W . We first present simple baseline models to generate points before considering network-generation processes based on preferential attachment . Motivated by empirical facts presented later , we seek to generate an embedding space such that entities with similar properties will be closer with each other compared with more distant points.1 0.0 0.2 0.4 0.6 0.8 No rm al ize d Po we r A ) Generating nodes of a network . A1 ) Gaussian model . A simple way to generate an embeddinglike space creates each point ⟨pi , wi⟩ where wi is from a standard Gaussian distribution N ( 0 , Id ) and the prominence p is drawn uniformly from a random permutation of { 1 , 2 , . . . , n } so that each entity i has a random rank pi . The right figure illustrates generated embedding points that are from 3-dimensional Gaussian distribution , i.e. , N ( 0 , I3 ) . We color each point i by its normalized prominence , i.e. , pi/maxi∈V pi . As expected , these data points form a single cluster without any finer substructure within it . A2 ) Preferential Placement ( PP ) model . This model presumes that new entities emerge around the older , relatively more prominent ones . This PP model builds on the preferential attachment mechanism driving the development of real-world networks and graphs , where the frequency of generated entities exhibit the power law distribution , and it is closest to Simon ( 1955 ) .2 1We describe experimental details of generating points in Appendix A.1 . 2The idea can be traced back to Eggenberger & Pólya ( 1923 ) and Yule ( 1925 ) , with accepted modern formulations due to Simon ( 1955 ) , Price ( 1976 ) and Barabási & Albert ( 1999 ) . PP is parameterized by the total frequencies of data points n , the probability of increasing the frequency of an old entity q , and an exponential distribution3 with scale parameter β driving the distance of a new entity from the old one per-iteration t. The PP model randomly selects entity i from existing ones with a probability proportional to their frequencies and with a probability q , it increases the frequency of wi by 1 . Other- wise , it generates a new entity wt within a small ball around wi : wt = wi + rt · xt with pt = 1 , where the radius rt ∼ exp ( β ) , and its direction xt follows a uniform distribution on a unit sphere.4 Different from the above Gaussian model , the left figure shows generated points where more prominent points in the small core cluster are surrounded by less prominent ones . A3 ) Directional Preferential Placement ( DPP ) model . The previous PP model generates new points in uniformly distributed random directions , producing a homogeneous scattering of embedding entities . To bring about a more heterogeneous landscape with cluster structure , we use the von Mises-Fisher distribution ( vMF ) that allows more control over the directions where new entities are generated , in particular making this process more spatial context-aware . Specifically , a random unit vector x ∈ Rd , d ≥ 2 , follows the d-dimensional vMF distribution if its probability density function is given by fvMF ( x|µ , κ ) : = κd/2−1 ( 2π ) d/2Id/2−1 ( κ ) exp { κµ⊤x } , where µ with ∥µ∥ = 1 , is the mean direction parameter specifying the orientation relatively to the origin ; κ ≥ 0 is a concentration parameter controlling the dispersion about the mean ( higher κ produces higher concentration ) ; and Id/2−1 ( κ ) is the modified Bessel function of the first kind . For example , mean direction may represent the force attracting x to some local point , or , say , repelling it from a global center . vMF distribution has been applied for high-dimensional directional data analysis , particularly in applications to textual data and with a focus on clustering ( Banerjee et al . ( 2005 ) and Gopal & Yang ( 2014 ) , also see Sra ( 2018 ) for a recent overview ) . However , to the best of our knowledge , prior work considered some given fixed parameters of interest ( usually focusing on estimating them , sometimes on using them for generating exercises ) ; while this paper is interested exclusively in data-generating models , and treats at least some of these parameters ( e.g. , mean direction µ ) as endogenously defined within the model itself . In the DPP model , the first entity ⟨p0 , w0⟩ is initialized with p0 = 1 and ∥w0∥ = 1 . Given the total frequencies of entities n , the exponential distribution exp ( β ) with parameter β , and the probability of increasing the frequency of an old entity q , the DPP model involves ( 1 ) selection as an origin of an existing entity wi randomly with a probability proportional to its frequency ; and then ( 2 ) at each iteration t , with probability q , either ( 2a ) an increase of the frequency of the chosen wi by 1 ; or ( 2b ) a creation of a new entity around the origin wi , i.e. , wt = wi + rt ·xt . In contrast to the previous model , however , now the direction is distributed as xt ∼ vMF ( x|µt , κ ) , with mean direction vector µt pointing away from the global center of mass w∗t . 5 The generated points of this model is illustrated on figure to the right where higher weight nodes are close to each other .

Summary: This paper tries to explore the natural processes that generate new knowledge or concepts. Specifically, the authors propose two metrics to characterize embeddings trained on different datasets. Then, they compare some synthetic data generated by models and real-world data according to the aforementioned metrics. Finally, they conclude that the real-world data can be well simulated by a certain generative model.