Revealing interpretable object representations from human behavior

To study how mental object representations are related to behavior , we estimated sparse , non-negative representations of objects using human behavioral judgments on images representative of 1,854 object categories . These representations predicted a latent similarity structure between objects , which captured most of the explainable variance in human behavioral judgments . Individual dimensions in the low-dimensional embedding were found to be highly reproducible and interpretable as conveying degrees of taxonomic membership , functionality , and perceptual attributes . We further demonstrated the predictive power of the embeddings for explaining other forms of human behavior , including categorization , typicality judgments , and feature ratings , suggesting that the dimensions reflect human conceptual representations of objects beyond the specific task . 1 INTRODUCTION . A central goal in understanding the human mind is to determine how object concepts are represented and how they relate to human behavior . Given the near-infinite number of tasks or contexts of usage , this might appear to be an impossible prospect . Take the example of picking a tomato in a grocery store . A person may recognize it by its color , shape , size , or texture ; they may also know many of the more conceptual dimensions of a tomato , such as it being a fruit , or functional ones such as being a salad item . In most cases , however , only a few of these aspects might matter , depending on task context ( e.g . as a grocery item or as a projectile in an audience that is not pleased with a presentation ) . Thus , to understand how we interact meaningfully with the objects around us , we need to tackle three problems simultaneously : ( 1 ) determine the object concept , the cognitive representation of behaviorally-relevant dimensions that distinguish an object from other objects . ( 2 ) determine which object concept representations are integrated into our perceptual decisions . ( 3 ) characterize the influence of the context – determined by the surrounding objects – on those decisions . There have been many attempts to represent object concepts in terms of semantic features , a vector of variables indicating the presence of different aspects of their meaning . These representations have been used to model phenomena such as judgments of typicality or similarity between concepts , or reaction times in various semantic tasks , with the goal of drawing conclusions about mental representations of those concepts ( see Murphy ( 2004 ) for an extensive review of this literature ) . These features have usually been binary properties , postulated by researchers . A landmark study McRae et al . ( 2005 ) departed from this approach by instead asking hundreds of subjects to name binary properties of 541 objects , yielding 2,526 such semantic features . These corresponded to different types of information , e.g . for the concept `` hammer '' subjects might list taxonomic ( `` is a tool '' ) , functional ( `` is used to hammer nails '' ) , perceptual ( `` heavy '' ) , among others . The results revealed that concepts for objects in the same basic level category shared many features ; at the same time , there were also features that distinguished even very similar concepts . This effort was later replicated and extended by Devereux et al . ( 2014 ) , generating 5,929 semantic features for 638 objects . The main issues with features produced in either study are the lack of degree ( they can only be present or absent , albeit with varying naming frequencies ) , the extreme specificity of some features , and the omission of many features shared by the majority of concepts . A separate concern is the fact that , absent a specific context of use for each object , subjects will not likely think of many valid properties ( e.g . that a tomato can be thrown ) . A different approach is to postulate the existence of certain semantic features and ask subjects to judge the degree to which features are salient for each concept , instead of assuming binary features . Binder et al . ( 2016 ) did this for 65 features corresponding to types of information for which there is evidence of brain representation ( in their terminology : sensory , motor , spatial , temporal , affective , social , cognitive , etc ) . This requires experts to specify the features in advance , and not all features can be judged easily by salience . In all three approaches outlined above , there is also no clear way of determining which features are critical to semantic behavior . Here we introduce an approach that uses only information from behavioral judgments about the grouping of object images in the context of other objects to estimate representations of object concepts . We demonstrate that this approach can predict human behavior in face of new combinations of objects and that it also allows prediction of results in other behavioral tasks . We show that individual dimensions in the low-dimensional embedding represent complex combinations of the information in the binary features in Devereux et al . ( 2014 ) and , furthermore , are interpretable as conveying taxonomic , functional , or perceptual information . Finally , we will discuss the way in which this representation suggests a simple , effective model of judgments of semantic similarity in context . 2 DATA AND METHODS . 2.1 THE ODD-ONE-OUT DATASET . The collection of the Odd-one-out behavioral dataset is an ongoing project by the authors . The dataset contains information about 1,854 different object concepts , which we will denote by c1 , . . . , cm ( e.g . c1 = ‘ aardvark ’ , . . . , c1854 = ‘ zucchini ’ ) . A triplet consists of a set of three concepts { ci1 , ci2 , ci3 } , for instance , { c2 , c117 , c136 } = { ‘ abacus ’ , ‘ beetle ’ , ‘ bison ’ } , presented to Amazon Mechanical Turk ( AMT ) workers as three photographs of exemplars , one for each of the three concepts . Importantly , these photographs were accompanied neither by labels or captions , meaning behavior was based only on those pictures . Workers were asked to consider the three pairs within the triplet { ( ci1 , ci2 ) , ( ci1 , ci3 ) , ( ci2 , ci3 ) } , and to decide which item had the smallest similarity to the other two ( the `` odd-one-out '' ) . This is equivalent to choosing the pair with the greatest similarity . Let ( y1 , y2 ) denote the indices in this pair , e.g . for ‘ beetle ’ and ‘ bison ’ they would be ( y1 , y2 ) = ( 117 , 136 ) . The large number of objects in this dataset has two different motivations . The first is obtaining behavioral data about objects from a wide range of semantic categories ( tools , animals , insects , weapons , food , body parts , etc ) , rather than the much smaller numbers in prior studies . We will use the term `` category '' loosely , to refer to a taxonomic level above that of the individual object ; this is often , but not always , a basic level category ( Murphy , 2004 ) . The second is to have subjects consider each object in as many different contexts as possible . We take the contexts instantiated by comparing each object to the others in the triplet as a proxy of the many situations each object might appear or be used in . For instance , `` tomato '' might be paired with `` milk '' if the third object in the triplet were `` table '' ( `` tomato '' and `` milk '' are food ) , but perhaps not if the third object was `` wine '' ( `` milk '' and `` wine '' are a more specific kind of food ) . If `` tomato '' were presented with `` lipstick '' and `` rock '' , it could be paired with the former ( because they are red ) or the latter ( because they can both be projectiles ) . Finally , the presence of several objects sharing a given semantic category means that there is redundancy . Hence , one should be able to predict behavior in face of an object in novel contexts , if similar objects have already been observed in those contexts . The dataset in this paper contains judgments on 1,450,119 randomly selected triplets , roughly 0.13 % out of all possible triplets using the 1,854 concepts . These were the triplets remaining after excluding AMT participants that showed evidence of inappropriate subject behavior , namely if responses were unusually fast , exhibited systematic response bias , or were empty ( 137,281 triplets or 8.65 % ) . We plan to collect additional triplets and release all data by the end of the study . 2.2 SPARSE POSITIVE SIMILARITY EMBEDDING ( SPOSE ) . Semantic representation of concepts Our method , Sparse Positive Similarity Embedding ( SPoSE ) is based on three assumptions . The first is that each object concept ci can be represented by semantic features , quantified in a vector xi = ( xi1 , . . . , xip ) . These features represent aspects of meaning of the concept known by subjects , and correspond to dimensions of a space where the geometrical relationships between vectors encode the ( dis ) similarity of the corresponding concepts . We estimate semantic features from behavioral data alone , using the probabilistic model described in the next section . In our model , each feature/dimension xif in the vector xi is real and non-negative , so as to make it interpretable as the degree to which the aspect of meaning it represents is present and influences subject behavior ( Murphy et al. , 2012 ) . Further , we expect features/dimensions to be sparse McRae et al . ( 2005 ) , which led us to add a sparsity parameter to our model . Based on related work in sparse positive word embeddings of words from text corpora ( Murphy et al. , 2012 ) , we expect some of the dimensions to indicate membership of the semantic category the object belongs to ( e.g . tool , vegetable , vehicle , etc ) . If two objects in a triplet share a category and the third does not , subjects will tend to group objects with a common category with very high probability . However , categories alone can not fully explain subject performance in this task , as there may be triplets where either ( a ) all three objects belong to the same category or ( b ) no two objects share a category . We expect our method to find non-taxonomic information similar to that in McRae et al . ( 2005 ) and Devereux et al . ( 2014 ) , if it allow us to explain semantic decision making behavior . Probabilistic model Our second assumption is that the decision in a given trial is explained as a function of the similarity between the embedding vectors of the three concepts presented . Given two concepts ci and cj , the similarity Sij is the dot product of the corresponding vectors xi and xj : Sij = 〈xi , xj〉 = p∑ f=1 xifxjf . ( 1 ) Our third assumption is that the outcome of the triplet task is stochastic . This assumption reflects both trial-to-trial inconsistencies in how an individual subject makes the decision , as well as subject differences in the decision-making process . Given that we pool data across many subjects , we do not distinguish between these two sources of randomness . We model the probability that a subject will choose a given pair out of the three possible pairs as proportional to the exponential of the similarity , Pr [ y1 , y2 ] ∝ exp ( Sy1 , y2 ) . Since probabilities of the three possible outcomes add to one , this gives Pr [ y1 , y2 ] = exp ( Sy1 , y2 ) exp ( Si1 , i2 ) + exp ( Si1 , i3 ) + exp ( Si2 , i3 ) . ( 2 ) The vectors xi in this model are not normalized , in that each dimension – and , therefore , vector similarity – can be arbitrarily large . In addition to the reasons described in the previous section , this allows us to model situations where the decision is close to being deterministic ( e.g . two concepts are so similar that they will always be grouped together , regardless of third concept ) . We also tried modeling choice probabilities using Euclidean distances ( rather than dot product ) between embedding vectors , but this resulted in similar ( but overall slightly worse ) results on our evaluations . Parameter fitting We infer embedding vectors ( x1 , . . . , xm ) from data by fitting a regularized maximum likelihood objective function . Let n be the total number of triplets available for building the model . For the jth triplet in the dataset , let ( i1 , j , i2 , j , i3 , j ) denote the indices of the concepts in the triplet , and let ( y1 , j , y2 , j ) indicate the pair of concepts chosen , for triplet j = 1 , . . . , n. We randomly split the dataset into ntrain triplets for training and nval for choosing the sparsity regularization parameter λ . We index the training set by j = 1 , . . . , ntrain and the validation set by j = ntrain + 1 , . . . , n. The objective function for SPoSE , given a specified number of dimensions p , is ntrain∑ j=1 log ( exp ( xy1 , j Txy2 , j ) exp ( xi1 , j Txi2 , j ) + exp ( xi1 , j Txi3 , j ) + exp ( xi2 , j Txi3 , j ) ) + λ m∑ i=1 ||xi||1 ( 3 ) Here || · ||1 is the L1 norm , so ||x||1 = ∑p f=1 |xf | , and xf ≥ 0 for f = 1 , . . . , p. We use the Adam algorithm ( Kingma & Ba , 2015 ) with an initial learning rate of 0.001 to minimize the objective function , using a fixed number of 1,000 epochs over the training set , which was sufficient to ensure convergence . Since this objective function is non-convex , Adam is likely to find only a local minimum . However , the combination of non-negativity and L1 penalty leads to very similar solutions across random initializations ; we discuss this further in Section 3.1 . We select the regularization parameter λ out of a grid of candidate values by choosing the one that achieves the lowest cross-entropy CEv on the validation set , defined by CEv = n∑ j=ntrain+1 log ( exp ( xy1 , j Txy2 , j ) exp ( xi1 , j Txi2 , j ) + exp ( xi1 , j Txi3 , j ) + exp ( xi2 , j Txi3 , j ) ) . The dimensionality of the embedding , p , can be determined heuristically from the data . If p is set to be larger than the number of dimensions supported by the data , the SPoSE algorithm will shrink entire dimensions towards zero , where they can be easily thresholded . We believe this happens because the L1 penalty objective encourages redundant similar dimensions to be merged , which would not happen with an L2 penalty ; we provide a more formal justification in the Supplementary Material .

Summary: This is an interesting paper with a new approach to learn a sparse, positive (and hence interpretable) semantic space that maximizes human similarity judgements, by training to specifically maximize the prediction of human similarity judgements. The authors have collected the dataset themselves and have rating of sets of 3 objects from 1854 unique objects. They end up with a space (SPoSE) with relatively low dimensionality with respect to usual word embeddings (49 dimension) but perhaps not surprising when considering the small size of the words to embed. The authors run a set of experiment to show the usefulness of SPoSE. The most interesting one is the prediction of its dimensions by the CSLB features, which reveals a nice clustering in the different SPoSE dimensions. Perhaps the results would be a little more convincing if additional common word embeddings were also tested.