VICE: Variational Inference for Concept Embeddings

1 INTRODUCTION AND RELATED WORK . Human knowledge about object concepts encompasses many types of information , ranging from function to visual appearance , as well as encyclopedic facts or taxonomic characteristics . This knowledge supports the identification of objects , inferences about what interactions they support , or what the effects of such interactions in the environment will be . Key questions for cognitive scientists modelling human performance in experiments are 1 ) which of this information is accessible to participants and 2 ) how is it used across different tasks . Several studies ( McRae et al. , 2005 ; Devereux et al. , 2013 ; Buchanan et al. , 2019 ; Hovhannisyan et al. , 2021 ) have asked subjects to list properties for hundreds to thousands of objects , yielding thousands of answers about the types of information above . Properties exist at many levels , ranging from categorization ( e.g . `` is an animal '' ) to very specific facts ( e.g . `` is eaten in France '' ) . Objects are implicitly represented as a vector of binary properties . This approach is agnostic to downstream prediction tasks , but does not provide an indication of which properties are more important – other than frequency of listing – and does not allow for graded property values . An alternative approach is for researchers to postulate dimensions of interest , and then ask human subjects to place each object in each dimension . An example is Binder et al . ( 2016 ) , who collected ratings for hundreds of objects , as well as verbs and adjectives , in 65 dimensions reflecting sensory , motor , spatial , temporal , affective , social , and cognitive experiences . The overall problem is then one of discovering a representation for objects that is not biased by a particular task , and is interpretable without requiring researchers to postulate the types of information represented . Several researchers have tried to develop interpretable concept representation spaces from text corpora , via word embeddings with positivity and sparsity constraints ( Murphy et al. , 2012 ) , topic model representations of Wikipedia articles about objects ( Pereira et al. , 2013 ) , transformations of word embeddings into sparse , positive spaces ( Subramanian et al. , 2018 ; Panigrahi et al. , 2019 ) or predictions of properties ( Devereux et al. , 2013 ) or dimensions ( Utsumi , 2020 ) , or text corpora combined with imaging data ( Fyshe et al. , 2014 ) or with object images ( Derby et al. , 2018 ) . Finally , Derby et al . ( 2019 ) introduced a neural network mapping the sparse feature space of a semantic property norm to the dense space of a word embedding , identifying informative combinations of properties or allowing ranking of candidate properties for arbitrary words . Recently , Zheng et al . ( 2019 ) and Hebart et al . ( 2020 ) introduced SPoSE , a model of the mental representations of 1,854 objects in a 49-dimensional space . The model was derived from a dataset of 1.5M Amazon Mechanical Turk ( AMT ) judgments of object similarity , where subjects were asked which of a random triplet of objects was the odd one out . The model embedded each object as a vector in a space where each dimension was constrained to be sparse and positive . Triplet judgments were predicted as a function of the similarity between embedding vectors of the three objects considered . The authors showed that these dimensions were predictable as a combination of elementary properties in the Devereux et al . ( 2013 ) norm , which often co-occur across many objects . Hebart et al . ( 2020 ) further showed that 1 ) human subjects could coherently label what the dimensions were “ about ” , ranging from categorical ( e.g . is animate , food , drink , building ) to functional ( e.g . container , tool ) or structural ( e.g . made of metal or wood , has inner structure ) . Subjects could also predict what dimension values new objects would have , based on knowing the dimension value for a few other objects . These results suggest that SPoSE captures core object knowledge that subjects use . Navarro & Griffiths ( 2008 ) introduced a related method for learning semantic concept embeddings from similarity data , which infers the number of latent dimensions using the Indian Buffet Process ( IBP , Griffiths & Ghahramani ( 2011 ) ) , but their approach is not directly applicable to our setting due to reliance on continuous-valued similarity ratings instead of forced-choice behavior . Furthermore , it is known to be challenging to scale the IBP to the number of features and observations considered in our work ( Ghahramani , 2013 ) . Roads & Love ( 2021 ) introduced a related method for deriving an object embedding from behavior in a 8-rank-2 task . Their method aimed to predict behavior from the embeddings , using active sampling to query subjects with the most informative stimuli . The method was not meant to produce interpretable dimensions , but rather construct object similarity matrix as efficiently as possible . There is growing interest by cognitive scientists in using SPoSE , as it makes it possible to discover an item representation for any kind of item amenable to an odd-one-out comparison in a triplet task . Furthermore , the combination of positivity and sparsity constraints in each dimension of the representation leads to interpretability by human subjects : no item is represented by every dimension , and most dimensions are present for only a few items . That item representation can then be used within other behavioral prediction models , to make predictions about neuroimaging data , etc . For this potential to be realized , however , we believe a number of issues with SPoSE should be addressed . The first is the use of an l1 sparsity penalty to promote interpretability of dimensions . l1 achieves sparsity at the cost of unnecessarily shrinking larger values ( Belloni & Chernozhukov , 2013 ) . In SPoSE , 6-11 dominant dimensions for an object account for most of the prediction performance ; the cost of removing irrelevant dimensions is to potentially make dominant dimensions smaller than they should be , and affect performance . Second , the l1 penalty is analogous to having a Laplace prior over those values . If we consider the distribution of values across objects for the two most important SPoSE dimensions , in Figure 1 , we can see that they have a bimodal distribution , with a spike around 0 and a much smaller , wide slab of probability for non-zero values , which is not Laplace . Overcoming this `` wrong '' prior requires more data than strictly necessary to learn the representation . SPoSE was developed with a dataset that was orders of magnitude larger than what a typical experiment might collect , but it was never tested on smaller datasets . Finally , SPoSE uses a heuristic , subjective criterion for determining how many dimensions the solution should have . In this paper we introduce VICE , an approach for variational inference of object concept embeddings in a space with interpretable sparse , positive dimensions , which addresses the SPoSE issues identified above . Specifically , we encourage sparsity and small weights by using a spike-and-slab prior . This is more appropriate than a Laplace prior , because importance – the value an object takes in a dimension – is different from relevance – whether the dimension matters for that object – and they can be controlled separately with a spike-and-slab prior . The prior hyperparameters are meant to be intuitive to a user , and to make it easier to specify hypotheses about dimensional structure . We use variational Bayes both because it is a Bayesian approach , and also because it assumes a unimodal posterior for the loading of each object in each dimension . It also allows a more principled procedure for determining how many dimensions the model should have , by taking into account uncertainty about their values . We compare our model with SPoSE over different subsets of the dataset used to develop it , and verify that it performs as well or better by various criteria : prediction of behavior , calibration of the prediction of decision probabilities , and reproducibility of solutions across seeds . Importantly , it has significantly better performance on smaller datasets ( 5− 10 % of the original SPoSE dataset ) . Our implementation of VICE is available on GitHub1 , and will be de-anonymized upon acceptance . 2 METHODS . 2.1 ODD-ONE-OUT TASK . The odd-one-out task is motivated by the problem of discovering object embeddings based on similarity judgments involving a set of m different object concepts , which we will denote by c1 , . . . , cm ( e.g . c1 = ‘ aardvark ’ , . . . , c1854 = ‘ zucchini ’ ) . These similarity judgments are collected from human participants , who are given queries which consist of a ‘ triplet ‘ of three concepts { ci1 , ci2 , ci3 } , for instance , { c268 , c609 , c1581 } = { ‘ suit ’ , ‘ flamingo ’ , ‘ car ’ } . Participants are asked to consider the three pairs within the triplet { ( ci1 , ci2 ) , ( ci1 , ci3 ) , ( ci2 , ci3 ) } , and to decide which item had the smallest similarity to the other two ( the `` odd-one-out '' ) . This is equivalent to choosing the pair with the greatest similarity . Let ( y1 , y2 ) denote the indices in this pair , e.g . for ‘ suit ’ and ‘ flamingo ’ they would be ( y1 , y2 ) = ( 268 , 609 ) . A dataset D is a set of N pairs of concept triplets and one-hot vectors that correspond to the index two most similar concepts , i.e . ( { ci1 , ci2 , ci3 } , ( yi1 , yi2 ) ) . 2.2 SPOSE . Sparse Positive object Similarity Embedding ( SPoSE ) ( Zheng et al. , 2019 ) is an approach for finding interpretable item dimensions from an odd-one-out task . It does so by finding an embedding vector xi = ( xi1 , . . . , xip ) for every item ci . The similarity Sij of two items ( e.g . ci and cj ) is computed by the dot product of the corresponding embeddings ( i.e . xi and xj ) , Sij = 〈xi , xj〉 From these similarities , the probability of choosing ( yi1 , yi2 ) as the most similar pair of items given the item triplet { ci1 , ci2 , ci3 } and given embedding vectors { xi1 , xi2 , xi3 } is computed as : p ( ( yi1 , yi2 ) | { ci1 , ci2 , ci3 } , { xi1 , xi2 , xi3 } ) = exp ( Syi1 , yi2 ) exp ( Si1 , i2 ) + exp ( Si1 , i3 ) + exp ( Si2 , i3 ) . ( 1 ) SPoSE uses a maximum a posterori ( MAP ) estimation to find the most likely embedding given the training data and a prior : argmax X log p ( X|Dtrain ) = argmax X log p ( Dtrain|X ) + log p ( X ) , ( 2 ) 1Link to anonymous GitHub repository : https : //anonymous.4open.science/r/VICE-59F0 where X is a matrix containing the embedding vectors for all of the items and p ( X ) is a prior for the embeddings , and p ( Dtrain , j |X ) is defined in ( 7 ) . To induce sparsity in the embeddings , SPoSE uses a mean-field Laplace prior , leading to this objective : argmax X ntrain∑ j=1 log p ( Dtrain , j |X ) + λ m∑ i=1 ||xi||1 ( 3 ) Here , || · ||1 is the l1 norm , so ||x||1 = ∑p f=1 |xf | , and xf ≥ 0 for f = 1 , . . . , p. The regularization parameter , λ , is selected out of a grid of candidate values by choosing the one that achieves the lowest ( average ) cross-entropy on the validation set ( across twenty random seeds ) . The final dimensionality of the embedding , p , is determined heuristically from the data . If p is set to be larger than the number of dimensions supported by the data , the SPoSE algorithm will shrink entire dimensions towards zero by removing weights with a magnitude less than a given absolute threshold . While a threshold of 0.1 is suggested ( Zheng et al. , 2019 ) , no justification is given for that particular value , which is problematic given that the number of dimensions removed is quite sensitive to that choice .

Summary: This paper presents a technique for learning image embeddings from similarity data provided as odd-one-out judgments over triplets of images (i.e., ball is more similar to apple than car). The authors build on an earlier technique called SPoSE that learns sparse, non-negative embeddings for images by maximizing the probability of choosing the right pair (where this similarity is calculated using the dot-product of image embeddings) with an L1 penalty on embeddings. In this paper, the authors argue that 1) a spike and slab prior is more suited for this setting and 2) a more principled approach to choosing the number of dimensions in learned embeddings is possible.  They present an improved version of SpoSE (called VICE) that uses variational inference to learn embeddings with uncertainties associated with them and place a spike-and-slab prior on the embeddings to encourage sparsity. They then present a way to prune the set of learned dimensions. This calculates the importance of each dimension first (using the learned uncertainties and false discovery rate), then clusters these, and finds the subset of clusters that give the best validation performance.  They argue that their technique is more principled than SpOSE and show that it performs similarly to it in terms of prediction accuracy. However, in small data regime, their model outperforms SpOSE.