VICE: Variational Inference for Concept Embeddings

1 INTRODUCTION AND RELATED WORK . Human knowledge about object concepts encompasses many types of information , ranging from function to visual appearance , as well as encyclopedic facts or taxonomic characteristics . This knowledge supports the identification of objects , inferences about what interactions they support , or what the effects of such interactions in the environment will be . Key questions for cognitive scientists modelling human performance in experiments are 1 ) which of this information is accessible to participants and 2 ) how is it used across different tasks . Several studies ( McRae et al. , 2005 ; Devereux et al. , 2013 ; Buchanan et al. , 2019 ; Hovhannisyan et al. , 2021 ) have asked subjects to list properties for hundreds to thousands of objects , yielding thousands of answers about the types of information above . Properties exist at many levels , ranging from categorization ( e.g . `` is an animal '' ) to very specific facts ( e.g . `` is eaten in France '' ) . Objects are implicitly represented as a vector of binary properties . This approach is agnostic to downstream prediction tasks , but does not provide an indication of which properties are more important – other than frequency of listing – and does not allow for graded property values . An alternative approach is for researchers to postulate dimensions of interest , and then ask human subjects to place each object in each dimension . An example is Binder et al . ( 2016 ) , who collected ratings for hundreds of objects , as well as verbs and adjectives , in 65 dimensions reflecting sensory , motor , spatial , temporal , affective , social , and cognitive experiences . The overall problem is then one of discovering a representation for objects that is not biased by a particular task , and is interpretable without requiring researchers to postulate the types of information represented . Several researchers have tried to develop interpretable concept representation spaces from text corpora , via word embeddings with positivity and sparsity constraints ( Murphy et al. , 2012 ) , topic model representations of Wikipedia articles about objects ( Pereira et al. , 2013 ) , transformations of word embeddings into sparse , positive spaces ( Subramanian et al. , 2018 ; Panigrahi et al. , 2019 ) or predictions of properties ( Devereux et al. , 2013 ) or dimensions ( Utsumi , 2020 ) , or text corpora combined with imaging data ( Fyshe et al. , 2014 ) or with object images ( Derby et al. , 2018 ) . Finally , Derby et al . ( 2019 ) introduced a neural network mapping the sparse feature space of a semantic property norm to the dense space of a word embedding , identifying informative combinations of properties or allowing ranking of candidate properties for arbitrary words . Recently , Zheng et al . ( 2019 ) and Hebart et al . ( 2020 ) introduced SPoSE , a model of the mental representations of 1,854 objects in a 49-dimensional space . The model was derived from a dataset of 1.5M Amazon Mechanical Turk ( AMT ) judgments of object similarity , where subjects were asked which of a random triplet of objects was the odd one out . The model embedded each object as a vector in a space where each dimension was constrained to be sparse and positive . Triplet judgments were predicted as a function of the similarity between embedding vectors of the three objects considered . The authors showed that these dimensions were predictable as a combination of elementary properties in the Devereux et al . ( 2013 ) norm , which often co-occur across many objects . Hebart et al . ( 2020 ) further showed that 1 ) human subjects could coherently label what the dimensions were “ about ” , ranging from categorical ( e.g . is animate , food , drink , building ) to functional ( e.g . container , tool ) or structural ( e.g . made of metal or wood , has inner structure ) . Subjects could also predict what dimension values new objects would have , based on knowing the dimension value for a few other objects . These results suggest that SPoSE captures core object knowledge that subjects use . Navarro & Griffiths ( 2008 ) introduced a related method for learning semantic concept embeddings from similarity data , which infers the number of latent dimensions using the Indian Buffet Process ( IBP , Griffiths & Ghahramani ( 2011 ) ) , but their approach is not directly applicable to our setting due to reliance on continuous-valued similarity ratings instead of forced-choice behavior . Furthermore , it is known to be challenging to scale the IBP to the number of features and observations considered in our work ( Ghahramani , 2013 ) . Roads & Love ( 2021 ) introduced a related method for deriving an object embedding from behavior in a 8-rank-2 task . Their method aimed to predict behavior from the embeddings , using active sampling to query subjects with the most informative stimuli . The method was not meant to produce interpretable dimensions , but rather construct object similarity matrix as efficiently as possible . There is growing interest by cognitive scientists in using SPoSE , as it makes it possible to discover an item representation for any kind of item amenable to an odd-one-out comparison in a triplet task . Furthermore , the combination of positivity and sparsity constraints in each dimension of the representation leads to interpretability by human subjects : no item is represented by every dimension , and most dimensions are present for only a few items . That item representation can then be used within other behavioral prediction models , to make predictions about neuroimaging data , etc . For this potential to be realized , however , we believe a number of issues with SPoSE should be addressed . The first is the use of an l1 sparsity penalty to promote interpretability of dimensions . l1 achieves sparsity at the cost of unnecessarily shrinking larger values ( Belloni & Chernozhukov , 2013 ) . In SPoSE , 6-11 dominant dimensions for an object account for most of the prediction performance ; the cost of removing irrelevant dimensions is to potentially make dominant dimensions smaller than they should be , and affect performance . Second , the l1 penalty is analogous to having a Laplace prior over those values . If we consider the distribution of values across objects for the two most important SPoSE dimensions , in Figure 1 , we can see that they have a bimodal distribution , with a spike around 0 and a much smaller , wide slab of probability for non-zero values , which is not Laplace . Overcoming this `` wrong '' prior requires more data than strictly necessary to learn the representation . SPoSE was developed with a dataset that was orders of magnitude larger than what a typical experiment might collect , but it was never tested on smaller datasets . Finally , SPoSE uses a heuristic , subjective criterion for determining how many dimensions the solution should have . In this paper we introduce VICE , an approach for variational inference of object concept embeddings in a space with interpretable sparse , positive dimensions , which addresses the SPoSE issues identified above . Specifically , we encourage sparsity and small weights by using a spike-and-slab prior . This is more appropriate than a Laplace prior , because importance – the value an object takes in a dimension – is different from relevance – whether the dimension matters for that object – and they can be controlled separately with a spike-and-slab prior . The prior hyperparameters are meant to be intuitive to a user , and to make it easier to specify hypotheses about dimensional structure . We use variational Bayes both because it is a Bayesian approach , and also because it assumes a unimodal posterior for the loading of each object in each dimension . It also allows a more principled procedure for determining how many dimensions the model should have , by taking into account uncertainty about their values . We compare our model with SPoSE over different subsets of the dataset used to develop it , and verify that it performs as well or better by various criteria : prediction of behavior , calibration of the prediction of decision probabilities , and reproducibility of solutions across seeds . Importantly , it has significantly better performance on smaller datasets ( 5− 10 % of the original SPoSE dataset ) . Our implementation of VICE is available on GitHub1 , and will be de-anonymized upon acceptance . 2 METHODS . 2.1 ODD-ONE-OUT TASK . The odd-one-out task is motivated by the problem of discovering object embeddings based on similarity judgments involving a set of m different object concepts , which we will denote by c1 , . . . , cm ( e.g . c1 = ‘ aardvark ’ , . . . , c1854 = ‘ zucchini ’ ) . These similarity judgments are collected from human participants , who are given queries which consist of a ‘ triplet ‘ of three concepts { ci1 , ci2 , ci3 } , for instance , { c268 , c609 , c1581 } = { ‘ suit ’ , ‘ flamingo ’ , ‘ car ’ } . Participants are asked to consider the three pairs within the triplet { ( ci1 , ci2 ) , ( ci1 , ci3 ) , ( ci2 , ci3 ) } , and to decide which item had the smallest similarity to the other two ( the `` odd-one-out '' ) . This is equivalent to choosing the pair with the greatest similarity . Let ( y1 , y2 ) denote the indices in this pair , e.g . for ‘ suit ’ and ‘ flamingo ’ they would be ( y1 , y2 ) = ( 268 , 609 ) . A dataset D is a set of N pairs of concept triplets and one-hot vectors that correspond to the index two most similar concepts , i.e . ( { ci1 , ci2 , ci3 } , ( yi1 , yi2 ) ) . 2.2 SPOSE . Sparse Positive object Similarity Embedding ( SPoSE ) ( Zheng et al. , 2019 ) is an approach for finding interpretable item dimensions from an odd-one-out task . It does so by finding an embedding vector xi = ( xi1 , . . . , xip ) for every item ci . The similarity Sij of two items ( e.g . ci and cj ) is computed by the dot product of the corresponding embeddings ( i.e . xi and xj ) , Sij = 〈xi , xj〉 From these similarities , the probability of choosing ( yi1 , yi2 ) as the most similar pair of items given the item triplet { ci1 , ci2 , ci3 } and given embedding vectors { xi1 , xi2 , xi3 } is computed as : p ( ( yi1 , yi2 ) | { ci1 , ci2 , ci3 } , { xi1 , xi2 , xi3 } ) = exp ( Syi1 , yi2 ) exp ( Si1 , i2 ) + exp ( Si1 , i3 ) + exp ( Si2 , i3 ) . ( 1 ) SPoSE uses a maximum a posterori ( MAP ) estimation to find the most likely embedding given the training data and a prior : argmax X log p ( X|Dtrain ) = argmax X log p ( Dtrain|X ) + log p ( X ) , ( 2 ) 1Link to anonymous GitHub repository : https : //anonymous.4open.science/r/VICE-59F0 where X is a matrix containing the embedding vectors for all of the items and p ( X ) is a prior for the embeddings , and p ( Dtrain , j |X ) is defined in ( 7 ) . To induce sparsity in the embeddings , SPoSE uses a mean-field Laplace prior , leading to this objective : argmax X ntrain∑ j=1 log p ( Dtrain , j |X ) + λ m∑ i=1 ||xi||1 ( 3 ) Here , || · ||1 is the l1 norm , so ||x||1 = ∑p f=1 |xf | , and xf ≥ 0 for f = 1 , . . . , p. The regularization parameter , λ , is selected out of a grid of candidate values by choosing the one that achieves the lowest ( average ) cross-entropy on the validation set ( across twenty random seeds ) . The final dimensionality of the embedding , p , is determined heuristically from the data . If p is set to be larger than the number of dimensions supported by the data , the SPoSE algorithm will shrink entire dimensions towards zero by removing weights with a magnitude less than a given absolute threshold . While a threshold of 0.1 is suggested ( Zheng et al. , 2019 ) , no justification is given for that particular value , which is problematic given that the number of dimensions removed is quite sensitive to that choice .

Summary: The paper address a problem in cognitive science: identify the embedding of objects in the brain's semantic space based on the subjective judgments of objects' similarity (odd-one-out task). It proposes a new model (VICE) to address a few issues of the previous model (SPoSE). The new method uses a different form of prior (spike-and-slab) for the embedding, variational Bayes with Gaussian posterior, and appear to perform as well as or better than SPoSE in several metrics