Visualizing and Understanding the Semantics of Embedding Spaces via Algebraic Formulae

1 INTRODUCTION . Learning representations is an important part of modern machine learning and natural language processing research . Those representations are often real-valued vectors also called embeddings and are obtained both as byproducts of supervised learning or as the direct goal of unsupervised methods . Independently of how the embeddings are learned , there is much value in understanding what information they capture , how they relate to each other and how the data they are learned from influences them . A better understanding of the embedded space may lead to a better understanding of the data , of the problem and the behavior of the model , and may lead to critical insights in improving such models . Because of their high-dimensional nature , they are hard to visualize effectively , and the most adopted approach is to project them in a bi-dimensional space . Projections have a few shortcomings : 1 ) they may not preserve distance in the original space , 2 ) they are not comparable across models and 3 ) do not provide interpretable dimensions of variability to project to , preventing for more detailed analysis and understanding . For these reasons , there is value in mapping embeddings into a more specific , controllable and interpretable semantic space . Principal Component Analysis ( PCA ) ( Pearson , 1901 ) and t-Distributed Stochastic Neighbor Embedding ( t-SNE ) ( van der Maaten & Hinton , 2008 ) are two projection techniques often used for visualizing embeddings in two dimensions , although other techniques can be used . PCA projects embeddings on a lower dimensional space that has the directions of the highest variance in the dataset as axes . Those dimensions do not carry any interpretable meaning , making interpretation difficult . By visualizing the first two dimensions of a PCA projection , the only insight obtainable is semantic relatedness ( Budanitsky & Hirst , 2006 ) between points by observing their relative closeness and therefore topical clusters can be identified . The downside is that embeddings that end up being close in the projected space may not be close in the original embedding space and vice versa . Moreover , as the directions of highest variance are different from embedding space to embedding space , the projections are incompatible among different embeddings spaces , and this makes them not comparable , a common issue among dimensionality reduction techniques . t-SNE , differently from PCA , optimizes a loss that encourages embeddings that are close in the original high-dimensional space to be close in the lower dimensional projection space . This helps in visualizing clusters better than with PCA , as t-SNE puts each point in the projected space so that distance in the original space with respect to its nearest neighbors is preserved as much as possible . Visualizations obtained in this way reflect more the original embedding space and topical clusters are more clearly distinguishable , but doesn ’ t solve the issue of comparability of two different sets of embeddings , nor it solves the lack of interpretability of the axes and still doesn ’ t allow for finegrained inspection . Moreover , t-SNE is pretty sensible to hyperparameters , making it unclear how much the projection reflects the data . In this paper , a new and simple method to inspect , explore and debug embedding spaces at a finegrained level is proposed . It consists in defining explicitly the axes of projection through formulae in vector algebra over the embeddings themselves . Explicit axis definition gives an interpretable and fine-grained semantics to the axes of projection . Defining axes explicitly makes it possible to analyze in a detailed way how embeddings relate to each other with respect to interpretable dimensions of variability , as carefully crafted formulas can map ( to a certain extent ) to semantically meaningful portions of the learned spaces . The explicit axes definition also allows for comparing of embeddings obtained from different datasets , as long as they have common labels . We demonstrate three visualizations for analyzing subspaces of interest of embedding spaces and a set of example case studies including bias detection , polysemy analysis and fine-grained embedding analysis . Additional tasks that may be performed using the proposed methodology and visualization are diachronic analysis and analysis of representations learned from graphs and knowledge bases . The proposed visualizations can moreover be used for debugging purposes and in general for obtaining a better understanding of the embedding spaces learned by different models and representation learning approaches . We are releasing an open-source 1 interactive tool that implements the proposed visualizations , in order to enable researchers in the fields of machine learning , computational linguistics , natural language processing , social sciences and digital humanities to perform exploratory analysis and better understand the semantics of their embeddings . The main contribution of this work lies in the use of explicit user-defined algebraic formulae as axes for projecting embedding spaces into semantically-meaningful subspaces that when visualized provide interpretable axes . We show how this methodology can be widely used through a series of case studies on well known models and data and we furthermore validate the how the visualizations are more interpretable through a user study . 2 RELATED WORK . 2.1 EMBEDDING METHODS AND APPLICATIONS . Several methods for learning embeddings from symbolic data have been recently proposed ( Pennington et al. , 2014 ; Mikolov et al. , 2013 ; Mnih & Kavukcuoglu , 2013 ; Lebret & Collobert , 2014 ; Ji et al. , 2016 ; Rudolph et al. , 2016 ; Nickel et al. , 2016 ) . The learned representations have been used for a variety of tasks like recommendation ( Barkan & Koenigstein , 2016 ) , link prediction on graphs ( Grover & Leskovec , 2016 ) , discovery of drug-drug interaction ( Abdelaziz et al. , 2017 ) and many more . In particular , positive results in learning embeddings for words using a surrogate prediction task ( Mikolov et al. , 2013 ) started the resurgence of interest in those methods , while a substantial body of research from the distributional semantics community using count and matrix factorization based methods ( Deerwester et al. , 1990 ; Baroni & Lenci , 2010 ; Kanerva et al. , 2000 ; Levy & Goldberg , 2014 ; Biemann & Riedl , 2013 ; Gabrilovich & Markovitch , 2007 ) was previously developed . Refer to Lenci ( 2018 ) for a comprehensive overview . 2.2 EMBEDDING VISUALIZATION . In their recent paper , Heimerl & Gleicher ( 2018 ) extracted a list of routinely conducted tasks where embeddings are employed in visual analytics for NLP , such as compare concepts , finding analogies , and predict contexts . iVisClustering ( Lee et al. , 2012 ) represents topic clusters as their most 1The tool will be made available after the review period to preserve double-blindness representative keywords and displays them as a 2D scatter plot and a set of linked visualization components supporting interactively constructing topic hierarchies . ConceptVector ( Park et al. , 2018 ) makes use of multiple keyword sets to encode the relevance scores of documents and topics : positive words , negative words , and irrelevant words . It allows users to select and build a concept iteratively . Liu et al . ( 2018 ) display pairs of analogous words obtained through analogy by projecting them on a 2D plane obtained through a PCA and an SVM to find the plane that separates words on the two sides of the analogy . Besides word embedding , textual visualization has been used to understand topic modeling ( Chuang et al. , 2012 ) and how topic models evolve over time ( Havre et al. , 2002 ) . Compared to literature , our work allows more fine-grained control over the conceptual axes and the filtering logic , e.g. , allowing users to define concept based on explicit algebraic formulae beyond single keywords ( Section 3 ) , metadata based filtering , as well as multidimensional and multi-data source comparison beyond the common 2D scatter plot view . ( Sec 4 ) 3 METHODOLOGY . The basic insight of this work is that goal-oriented inspection of embedding spaces can be defined in terms of items and dimensions of variability . For instance , if the goal is to discover if a dataset ( and by consequence an embedding model trained on it ) includes gender bias , a user may define professions as specific items of interest and analyze how they are distributed among the concept of “ male ” and “ female ” , the two dimensions of variability . We use this as a running example in this section , while in the next section we present how to turn goal definitions into visualizations . The dimensions of variability are defined as algebraic formulae that use embedding labels as atoms . Algebraic formulae are a composition vector math operators ( e.g. , add , sub , mul ) to be applied on vectors ( referenced by their label in the data , i.e . the vector of “ apple ” is obtained by using using the keyword “ apple ” in the formula ) . They are used as the axes of a subspace of the entire embedding space and can be interpreted as concepts . In our example we can define two axes amale = man and afemale = woman . These are the most simple formulae as they are made of only one literal , but any formula using algebraic operation can be used instead . For instance amale = man + him and afemale = woman + her could be used instead . Defining axes explicitly as algebraic formulae gives an interpretable semantics to the dimensions of variability and by consequence to the axes of the visualization . To project on the axes , different distance and similarity measures can be used ( euclidean distance , correlation , dot product ) , in particular we will use cosine similarity in the remaining of the paper , defined as cossim ( a , b ) = a·b‖a‖‖b‖ The items of the goal are a set defined by extention ( items = { item1 , . . . , itemn } ) or by intention ( with rules ) . The rules use items ’ embeddings or items ’ metadata , like word frequencies , parts of speech , sentiment or categories the label belongs to . Intentions identify a semantically coherent region of the embedding space through logical formulae . Rules using item ’ s embeddings are defined as re = 〈d , e , c , v〉 where d is a distance or similarity function , e is an algebraic formula that uses embeddings names as atoms and is resolved into a vector , c ∈ { < , ≤ , = , ≥ , > } , v is a numeric value . They can be used , for instance , to select all the items that have a d = cosinesimilarity with respect to e = job + profession that is c =≥ then v = 0.5 . Rules using item ’ s metadata instead use typed metadata associated with each item . An item can have categorical fields ( e.g. , words can be stop-words or not ) , set fields ( e.g. , the parts of speech a word can belongs to ) and numerical fields ( e.g. , unigram frequencies in a corpus ) associated with it . Rules are be defined as inclusion in a set : rm = icat ∩ set 6= ∅ where icat is the set of categories associated with an item , containing only one element in the case of categorical fields or multiple values in the case of set fields , while for numerical fields they are defined as ranges . Following on with our example , we can select some professions like “ doctor ” , “ nurse ” , “ teacher ” and “ astronaut ” as our items , or we can define the items of interest as the set of all words in the embedding space that are close to the word “ profession ” ( e.g. , cosine similarity greater than 0.7 ) , that are not too frequent ( inside a range of frequency from 100 to 1000 ) and that are not stop-words .

Summary: The idea of analyzing embedding spaces in a non-parametric (example-based) way is well-motivated. However, the main technical contribution of this paper is otherwise not clear - the methodology section covers a very broad set of techniques but doesn't provide a clear picture of what is novel; furthermore, it makes a strong assumption about linear structure in the embedding space that may not hold. (It's worth noting that t-SNE does not make this assumption.)