Year: 2021

Abstract:
In this paper we introduce Variational Inference for Concept Embeddings (VICE), a novel method for learning object concept embeddings from human behavior in an odd-one-out task. We use variational inference to obtain a sparse, non-negative solution, with uncertainty information about each embedding value. We leverage this information in a statistical procedure for selecting the dimensionality of the model, based on hypothesis-testing over a validation set. VICE performs as well or better than previous methods on a variety of criteria: accuracy of predicting human behavior in an odd-one-out task, calibration to (empirical) human choice probabilities, reproducibility of object representations across different random initializations, and superior performance on small datasets. The latter is particularly important in cognitive science, where data collection is expensive. Finally, VICE yields highly interpretable object representations, allowing humans to describe the characteristics being represented by each latent dimension.

heading: 1 INTRODUCTION AND RELATED WORK
text: Human knowledge about object concepts encompasses many types of information, ranging from function to visual appearance, as well as encyclopedic facts or taxonomic characteristics. This knowledge supports the identification of objects, inferences about what interactions they support, or what the effects of such interactions in the environment will be. Key questions for cognitive scientists modelling human performance in experiments are 1) which of this information is accessible to participants and 2) how is it used across different tasks. Several studies (McRae et al., 2005; Devereux et al., 2013; Buchanan et al., 2019; Hovhannisyan et al., 2021) have asked subjects to list properties for hundreds to thousands of objects, yielding thousands of answers about the types of information above. Properties exist at many levels, ranging from categorization (e.g. "is an animal") to very specific facts (e.g. "is eaten in France"). Objects are implicitly represented as a vector of binary properties. This approach is agnostic to downstream prediction tasks, but does not provide an indication of which properties are more important – other than frequency of listing – and does not allow for graded property values. An alternative approach is for researchers to postulate dimensions of interest, and then ask human subjects to place each object in each dimension. An example is Binder et al. (2016), who collected ratings for hundreds of objects, as well as verbs and adjectives, in 65 dimensions reflecting sensory, motor, spatial, temporal, affective, social, and cognitive experiences.
The overall problem is then one of discovering a representation for objects that is not biased by a particular task, and is interpretable without requiring researchers to postulate the types of information represented. Several researchers have tried to develop interpretable concept representation spaces from text corpora, via word embeddings with positivity and sparsity constraints (Murphy et al., 2012), topic model representations of Wikipedia articles about objects (Pereira et al., 2013), transformations of word embeddings into sparse, positive spaces (Subramanian et al., 2018; Panigrahi et al., 2019) or predictions of properties (Devereux et al., 2013) or dimensions (Utsumi, 2020), or text corpora combined with imaging data (Fyshe et al., 2014) or with object images (Derby et al., 2018). Finally, Derby et al. (2019) introduced a neural network mapping the sparse feature space of a semantic property norm to the dense space of a word embedding, identifying informative combinations of properties or allowing ranking of candidate properties for arbitrary words.
Recently, Zheng et al. (2019) and Hebart et al. (2020) introduced SPoSE, a model of the mental representations of 1,854 objects in a 49-dimensional space. The model was derived from a dataset of
1.5M Amazon Mechanical Turk (AMT) judgments of object similarity, where subjects were asked which of a random triplet of objects was the odd one out. The model embedded each object as a vector in a space where each dimension was constrained to be sparse and positive. Triplet judgments were predicted as a function of the similarity between embedding vectors of the three objects considered. The authors showed that these dimensions were predictable as a combination of elementary properties in the Devereux et al. (2013) norm, which often co-occur across many objects. Hebart et al. (2020) further showed that 1) human subjects could coherently label what the dimensions were “about”, ranging from categorical (e.g. is animate, food, drink, building) to functional (e.g. container, tool) or structural (e.g. made of metal or wood, has inner structure). Subjects could also predict what dimension values new objects would have, based on knowing the dimension value for a few other objects. These results suggest that SPoSE captures core object knowledge that subjects use. Navarro & Griffiths (2008) introduced a related method for learning semantic concept embeddings from similarity data, which infers the number of latent dimensions using the Indian Buffet Process (IBP, Griffiths & Ghahramani (2011)), but their approach is not directly applicable to our setting due to reliance on continuous-valued similarity ratings instead of forced-choice behavior. Furthermore, it is known to be challenging to scale the IBP to the number of features and observations considered in our work (Ghahramani, 2013). Roads & Love (2021) introduced a related method for deriving an object embedding from behavior in a 8-rank-2 task. Their method aimed to predict behavior from the embeddings, using active sampling to query subjects with the most informative stimuli. The method was not meant to produce interpretable dimensions, but rather construct object similarity matrix as efficiently as possible.
There is growing interest by cognitive scientists in using SPoSE, as it makes it possible to discover an item representation for any kind of item amenable to an odd-one-out comparison in a triplet task. Furthermore, the combination of positivity and sparsity constraints in each dimension of the representation leads to interpretability by human subjects: no item is represented by every dimension, and most dimensions are present for only a few items. That item representation can then be used within other behavioral prediction models, to make predictions about neuroimaging data, etc.
For this potential to be realized, however, we believe a number of issues with SPoSE should be addressed. The first is the use of an l1 sparsity penalty to promote interpretability of dimensions. l1 achieves sparsity at the cost of unnecessarily shrinking larger values (Belloni & Chernozhukov, 2013). In SPoSE, 6-11 dominant dimensions for an object account for most of the prediction performance; the cost of removing irrelevant dimensions is to potentially make dominant dimensions smaller than they should be, and affect performance. Second, the l1 penalty is analogous to having a Laplace prior over those values. If we consider the distribution of values across objects for the two most important SPoSE dimensions, in Figure 1, we can see that they have a bimodal distribution, with a spike around 0 and a much smaller, wide slab of probability for non-zero values, which is not Laplace. Overcoming this "wrong" prior requires more data than strictly necessary to learn the representation. SPoSE was developed with a dataset that was orders of magnitude larger than what a typical experiment might collect, but it was never tested on smaller datasets. Finally, SPoSE uses a heuristic, subjective criterion for determining how many dimensions the solution should have.
In this paper we introduce VICE, an approach for variational inference of object concept embeddings in a space with interpretable sparse, positive dimensions, which addresses the SPoSE issues identified above. Specifically, we encourage sparsity and small weights by using a spike-and-slab prior. This is more appropriate than a Laplace prior, because importance – the value an object takes in a dimension – is different from relevance – whether the dimension matters for that object – and they can be
controlled separately with a spike-and-slab prior. The prior hyperparameters are meant to be intuitive to a user, and to make it easier to specify hypotheses about dimensional structure. We use variational Bayes both because it is a Bayesian approach, and also because it assumes a unimodal posterior for the loading of each object in each dimension. It also allows a more principled procedure for determining how many dimensions the model should have, by taking into account uncertainty about their values. We compare our model with SPoSE over different subsets of the dataset used to develop it, and verify that it performs as well or better by various criteria: prediction of behavior, calibration of the prediction of decision probabilities, and reproducibility of solutions across seeds. Importantly, it has significantly better performance on smaller datasets (5− 10% of the original SPoSE dataset). Our implementation of VICE is available on GitHub1, and will be de-anonymized upon acceptance.

heading: 2 METHODS

heading: 2.1 ODD-ONE-OUT TASK
text: The odd-one-out task is motivated by the problem of discovering object embeddings based on similarity judgments involving a set of m different object concepts, which we will denote by c1, . . . , cm (e.g. c1 = ‘aardvark’,. . . , c1854 = ‘zucchini’). These similarity judgments are collected from human participants, who are given queries which consist of a ‘triplet‘ of three concepts {ci1 , ci2 , ci3}, for instance, {c268, c609, c1581} = {‘suit’, ‘flamingo’, ‘car’}. Participants are asked to consider the three pairs within the triplet {(ci1 , ci2), (ci1 , ci3), (ci2 , ci3)}, and to decide which item had the smallest similarity to the other two (the "odd-one-out"). This is equivalent to choosing the pair with the greatest similarity. Let (y1, y2) denote the indices in this pair, e.g. for ‘suit’ and ‘flamingo’ they would be (y1, y2) = (268, 609). A dataset D is a set of N pairs of concept triplets and one-hot vectors that correspond to the index two most similar concepts, i.e. ({ci1 , ci2 , ci3}, (yi1 , yi2)).

heading: 2.2 SPOSE
text: Sparse Positive object Similarity Embedding (SPoSE) (Zheng et al., 2019) is an approach for finding interpretable item dimensions from an odd-one-out task. It does so by finding an embedding vector xi = (xi1, . . . , xip) for every item ci. The similarity Sij of two items (e.g. ci and cj) is computed by the dot product of the corresponding embeddings (i.e. xi and xj), Sij = 〈xi,xj〉 From these similarities, the probability of choosing (yi1 , yi2) as the most similar pair of items given the item triplet {ci1 , ci2 , ci3} and given embedding vectors {xi1 , xi2 , xi3} is computed as:
p((yi1 , yi2)|{ci1 , ci2 , ci3}, {xi1 , xi2 , xi3}) = exp(Syi1 ,yi2 )
exp(Si1,i2) + exp(Si1,i3) + exp(Si2,i3) . (1)
SPoSE uses a maximum a posterori (MAP) estimation to find the most likely embedding given the training data and a prior:
argmax X log p(X|Dtrain) = argmax X log p(Dtrain|X) + log p(X), (2)
1Link to anonymous GitHub repository: https://anonymous.4open.science/r/VICE-59F0
where X is a matrix containing the embedding vectors for all of the items and p(X) is a prior for the embeddings, and p(Dtrain,j |X) is defined in (7). To induce sparsity in the embeddings, SPoSE uses a mean-field Laplace prior, leading to this objective:
argmax X ntrain∑ j=1 log p(Dtrain,j |X) + λ m∑ i=1 ||xi||1 (3)
Here, || · ||1 is the l1 norm, so ||x||1 = ∑p f=1 |xf |, and xf ≥ 0 for f = 1, . . . , p. The regularization parameter, λ, is selected out of a grid of candidate values by choosing the one that achieves the lowest (average) cross-entropy on the validation set (across twenty random seeds). The final dimensionality of the embedding, p, is determined heuristically from the data. If p is set to be larger than the number of dimensions supported by the data, the SPoSE algorithm will shrink entire dimensions towards zero by removing weights with a magnitude less than a given absolute threshold. While a threshold of 0.1 is suggested (Zheng et al., 2019), no justification is given for that particular value, which is problematic given that the number of dimensions removed is quite sensitive to that choice.

heading: 2.3 VICE

heading: 2.3.1 VARIATIONAL BAYESIAN INFERENCE
text: Given the goal of better approximating p(X|Dtrain), we use variational inference. We approximate p(X|Dtrain) with a variational distribution, qθ(X), where q is our chosen family of distributions, and θ is a parameter that is learned in order to optimize the Kullback–Leibler (KL) divergence to the true posterior, p(X|Dtrain). In variational inference, the KL divergence objective function is:
argmin θ Eqθ(X)
[ 1
ntrain (log qθ(X)− log p(X))−
1
ntrain ntrain∑ i=1 log p(Dtrain,i|X)
] (4)
In order to use variational inference, a parametric variational distribution must be chosen. For VICE, we use a Gaussian variational distribution with a diagonal covariance matrix qθ(X) = N (µ, diag(σ2)), where the learnable parameters θ are µ and σ. This means that each embedding dimension has a mean, the most likely value for that dimension, and a standard deviation, the propensity of the embedding value to be close to the mean.
Similarly to Blundell et al. (2015), we use a Monte Carlo (MC) approximation of the above objective function by sampling a limited number of Xs from qµ,σ(X) during training. We generate X by means of the reparameterization trick (Kingma & Welling, 2013), Xθ, = µ+ σ · where is an N × p matrix of standard normal variates, leading to the objective:
argmin θ
1
m m∑ j=1
[ 1
ntrain (log qθ(Xθ, j )− log p(Xθ, j ))−
1
ntrain ntrain∑ i=1 log p(Dtrain,i|[Xθ, j ]+ ] (5)
where j ∈ RN×p is entrywise N (0, 1) and where []+ is the ReLU function. As commonly done in the dropout and Bayesian neural network literature (Srivastava et al., 2014; Blundell et al., 2015; Gal & Ghahramani, 2016; McClure & Kriegeskorte, 2016), we set m to 1 for computational efficiency.
In Equation 5, the expected log-likelihood of the entire training data is computed. However, using the entire training data set to compute the gradient update often works poorly for non-convex objective functions. This is due to the expensive computational cost of each update and to the convergence to poorly generalizing solutions (Smith et al., 2020). As a result, we stochastically approximate (Robbins & Monro, 1951) the training log-likelihood using random subsets (i.e. mini-batches) of the training dataset, with each mini-batch consisting of b triplets. This leads to the final objective
argmin θ
1
ntrain (log qθ(Xθ, )− log p(Xθ, ))−
1
b b∑ i=1 log p(Dtrain,i|[Xθ, ]+)] (6)
recalling that
p(Dtrain,i|X) = exp(xy1,i
Txy2,i)
exp(xi1,i Txi2,i) + exp(xi1,i Txi3,i) + exp(xi2,j Txi3,i)
. (7)

heading: 2.3.2 SPIKE-AND-SLAB PRIOR
text: A key feature of SPoSE is sparsity. As discussed above, SPoSE induced sparsity using a zero-mean Laplace prior. We can empirically examine whether the Laplace prior is a realistic assumption, given the distribution of values in SPoSE dimensions. As Figure 1 depicts, these histograms do not resemble a Laplace distribution. Instead, it looks like there is a "spike" of probability at zero and a much smaller, but wide, "slab" of probability for the non-zero values of a SPoSE dimension. To model this, we use a spike-and-slab Gaussian mixture prior, as introduced in Blundell et al. (2015):
p(X) = N∏ i=1 p∏ f=1 (πN (xif ; 0, σ2spike) + (1− π)N (xif ; 0, σ2slab)) (8)
This prior has three parameters. π is the probability that an embedding dimension will be drawn from the spike Gaussian instead of the slab Gaussian. The standard deviations σspike and σslab control the likelihood of of an embedding value being set to 0 in the spike or slab distributions, respectively. xif is the embedding weight for the ith item in the f th dimension. Since spike and slab distributions are mathematically interchangeable, by convention we require that, σspike << σslab. In our experiments, these are chosen with grid search on one half of the validation set, the “tuning set”. (The other half of the validation set, the “pruning set”, is used for dimensionality reduction, as we describe in §2.3.4.)

heading: 2.3.3 PREDICTING THE ODD-ONE-OUT USING VICE
text: In this work, we consider two different prediction problems given a new triplet: (1) predicting the choice and (2) predicting the distribution of the choice. In either case, we start with computing the posterior probability distribution over the three triplet choices. If predicting a choice, then we output the choice with the maximum posterior probability. (For details on how we handle ties, see §3.3.1.) If the goal is to predict the distribution, then we return the predicted distribution.
The predicted probability distribution is computed from the variational posterior, qθ(X). When making predictions, we want to compute the probability of an odd-one-out for a given triplet. We approximate this probability by using an MC estimate fromm samplesXj = Xθ, j for j = 1, . . . ,m (Graves, 2011; Blundell et al., 2015; Kingma & Welling, 2014; McClure & Kriegeskorte, 2016; Blei et al., 2017). Mathematically, this means that we compute the predicted distribution as
p̂((yi1 , yi2)|{ci1 , ci2 , ci3}) ≈ 1
m m∑ j=1 p((yi1 , yi2)|{ci1 , ci2 , ci3}, {[x j i1 ]+, [x j i2 ]+, [x j i3 ]+}). (9)

heading: 2.3.4 DIMENSIONALITY REDUCTION FOR VICE
text: For interpretability purposes, it is crucial that the model does not use more dimensions than necessary. Zheng et al. (2019) accomplished this through the sparsity-inducing penalty that causes dimensions to shrink towards zero if they do not contribute to explaining the data. (see Section 2.2). Note, however, that these uninformative weights do not totally go to zero because of noise in the gradients. Hence, these dimensions were pruned by choosing a threshold for the L1 norm of the dimension based on looking at the “elbow plot” of the sorted L1 norms of the dimensions; this approach is subjective and highly dependent on the specific dataset. In VICE, the KL penalty we use has a similar effect of causing uninformative weights to shrink. Rather than using a user-defined threshold to prune dimensions, VICE exploits the uncertainty information obtained in training the model to select a set of informative dimensions. The pruning procedure consists of three steps: (1) assigning an importance score to each dimension; (2) clustering dimensions by importance; and (3) choosing the subset of clusters that best explains the validation set. We describe each of these three steps in detail below.
Assign an importance score to each dimension Intuitively, the importance score reflects the number of objects that we can confidently say have non-zero weight in a dimension. To compute the score, we start by using the variational embedding for each item i – location µij and scale σij parameters, to compute the posterior probability that the weight will be truncated to zero according to the left tail of a Gaussian distribution with that location and scale (as described in §2.3.1). This gives us a posterior probability of the weight taking the value zero for each item within a dimension (Graves, 2011). To calculate the overall importance of a dimension, we estimate the number of items that plausibly have non-zero weights given a user-specified False Discovery Rate target (FDR) (Benjamini & Hochberg, 1995). FDR provides a method for inferring the number of hypotheses which are non-null, based on an array of p-values, with statistical guarantees on the expected proportion of false rejections. We define dimension importance as the number of rejections given by the BH(q) algorithm, with the FDR tolerance q specified by the user, using the posterior zero-probabilities as the p-values.
Cluster dimensions by importance using a Gaussian mixture model Given the importance scores in the previous step, a reasonable approach would be to sort dimensions by importance, and then use the left-out half of the validation set, or “pruning set”, to determine the k most important dimensions to include. However, we found that this approach led to high variance, due to the existence of groups of dimensions with very similar importance scores. We hypothesized that these groups of dimension corresponded to different feature types, as observed in Zheng et al. (2019). As McRae et al. (2005) discusses, these features can be grouped into different feature types, such as categorical, functional, encyclopedic, visual-perceptual and non-visual-perceptual. Therefore, the second step in our pruning method creates clusters of dimensions that have similar importance. We fit GMMs with varied number of components k (e.g. k ∈ {1, 2, ..., 6}) to the importance scores for each dimension, and find the number of components/modes that show the lowest Bayesian Information Criterion (BIC). Here, we limit the number of possible clusters to 6, as a conservative estimate on the number of distinct feature types (e.g. categorical, functional, perceptual) with possibly differing sparsity ranges–i.e., categorical features may apply to a large subset of items, while specific visual features might apply only to a handful. We cluster dimensions into k modes.
Choosing the subset of dimension clusters that best explains the validation set We find the best non-empty subset of clusters of dimensions, in terms of cross-entropy on the validation “pruning” set, and prune all clusters of dimensions outside of this subset. (If a given feature is uninformative, then features with similar importance scores are likely to be similarly uninformative.)

heading: 3 EXPERIMENTS

heading: 3.1 DATA
text: We used two datasets from Zheng et al. (2019), selected after quality control. The first contained judgments on 1,450,119 randomly selected triplets. We used a random subsample of 90% of these triplets for the training set, and the remaining 10% for the validation set (tuning and pruning). The second was an independent test set of 19,968 triplets with 25 repeats for each of 1,000 randomly selected triplets; none of these were present in the training set. Having this many repeats allows us to be confident of the response probability for each triplet. Furthermore, it allows us to establish a model-free estimate of the Bayes accuracy, the best possible accuracy achievable by any model.

heading: 3.2 EXPERIMENTAL SETUP
text: Training We implemented both SPoSE and VICE in PyTorch (Paszke et al., 2019) using Adam (Kingma & Ba, 2015) with α = 0.001. To guarantee a fair comparison between VICE and SPoSE, each model configuration was trained using 20 different random seeds, for a fixed number of 1000 epochs. Each model was initialized with a weight matrix, W ∈ RD×N , where D was set to 100 and N refers to the number of unique items in the dataset (i.e., 1854). In preliminary experiments, we observed that, after pruning, no model was left with a latent space of more than 100 dimensions, which is why we did not consider models with higher initial dimensionality.
Other details Please see section §A.1 for weight initialization and hyperparameter tuning.

heading: 3.3 PREDICTION EXPERIMENTS

heading: 3.3.1 EVALUATION MEASURES
text: Prediction accuracy Since human triplet choices are represented as three-dimensional one-hotvectors, where 1 represents the odd-one-out choice for a particular triplet, it is simple to compare them with model choices. The choice of a model is computed as argmax p(ŷ|θ), where p(ŷ|θ) refers to a model’s softmax probability distribution over a triplet given the model parameters (see Equation 9). If there is a tie in the softmax output, we regard this as an incorrect choice. A model can either be correct or incorrect, and no partial credit is given, guaranteeing a conservative measure of a model’s prediction behavior. The reported prediction accuracy is the fraction of trials where the model predicted the correct odd-one-out item. We can get an estimated upper bound on the Bayes accuracy, i.e., the best possible accuracy of any model, by using the repeats in the independent test set. As the optimal model predicts the repeat majority outcome for any triplet, this accuracy ceiling – 0.673 – is the average probability of the majority outcome over the set of all triplets.
Predicting Human Uncertainty The triplet task is subjective: there is no correct answer to any given triplet, and often subjects give all three. The independent test set gives us the probability distribution over answers for each triplet, graded information about the relative similarities of the three item pairs. Predicting this distribution precisely is a more stringent test of model quality than prediction accuracy, and of even more relevance in cognitive science applications. We quantify this through the KL divergence between the softmax probabilities of a model (see Section 2.3.3) and the empirical human probability distributions, obtained by computing discrete probability distributions for triplet repeats on the independent test set (see Section 3.1). We use the KL divergence because it is a commonly used measure for assessing the similarity between two probability distributions.

heading: 3.3.2 EXPERIMENT RESULTS
text: Full dataset We compared pruned median models of VICE and SPoSE, where the median model was identified by the median cross-entropy error on the tuning set. For VICE, we set the number of MC samples to m = 50 (see Equation 9) . On the independent test set, VICE and SPoSE achieved a similarly high prediction accuracy of 0.6380 and 0.6378, respectively, versus a chance-level accuracy of 0.333(3). Likewise, VICE and SPoSE achieved similar KL-divergences of 0.103 and 0.105, respectively, versus a chance-level KL-divergence of 0.366. The differences between the median model predictions across individual triplets in the test set were not statistically significant, under the null hypothesis according to a two-sided paired t-test, for either accuracy or KL-divergence. Hence, VICE and SPoSE predicted triplets equally well when they were both trained on the full dataset. This is not surprising, as Bayesian methods based on MC sampling become more like deterministic Maximum Likelihood Estimation (MLE) the more training data is available, per the Bernstein-von Mises theorem (Doob, 1949). As a result, the effects of the prior are most prominent when models are trained on datasets where ntrain is not particularly large, as we will see in the next section.
Efficiency on smaller datasets Performance on small datasets is especially important in cognitive science, where behavioral experiments often have low sample sizes (e.g. tens to hundreds of volunteer
in-lab subjects ) or can be costly to scale in AMT. To test whether VICE can model the data better than SPoSE when data are scarce, we created non-overlapping subsets of the training dataset. Specifically, we did this for subsets with sizes equal to 5%, 10%, 20%, and 50% of the dataset, yielding 20, 10, 5, and 2 subsets, respectively. Validation and test sets were unchanged. In Figure 3, we show the average prediction accuracy and KL divergence across random seeds, for models trained on every dataset size, including the full training set. Averages were computed across both random seeds and training subsets, where the average over random seeds was identified first to get a per-subset estimate; performance across subsets was then used to compute the confidence intervals (CIs). Figure 3 shows that the difference in prediction accuracy and KL divergence between VICE and SPoSE became more pronounced the fewer triplet samples were used for training. The difference was striking for the 5% and 10% data subsets, with ≈ 67, 500 and ≈ 135, 000 triplets, respectively. In the former, SPoSE predicted at chance-level; in the latter, it showed a large variation between random seeds and data splits, as can be seen in the 95% CIs in Figure 3. In both low-resource scenarios, VICE showed a compellingly small variation in the two performance metrics across random seeds, and predicted much better than chance-level. The differences between VICE and SPoSE for the 5% and 10% subsample scenarios were statistically significant according to a two-sided paired t-test (p < 0.001), comparing individual triplet predictions between the pruned median models.

heading: 3.4 REPRODUCIBILITY EXPERIMENTS
text: Beyond predictive performance, a key criterion for learning concept representations is reproducibility, i.e., learning similar representations when using different random initializations on the same training data. To assess this, we compare 20 differently initialized VICE and SPoSE models.
The first aspect of reproducibility is finding similar numbers of dimensions, quantified as the standard deviation of that number across all 20 models. As shown in Table 1, VICE identified fewer dimensions than SPoSE, and this had a lower standard deviation across models (1.64 vs 2.30) The difference in standard deviation is, however, not statistically significant according to a two-sided F-test (F = 0.516, df = 19, p = 0.918). The second aspect is the extent to which the dimensions identified are similar across initializations. Since the embedding is not an ordered set of dimensions, we will deem a dimension learned in one VICE model reproducible if it is present in another independently trained instance of VICE, perhaps in a different column or with some small perturbation to the weights. To evaluate the number of highly reproducible dimensions we correspond each embedding dimension of a given initialization (after the pruning step) with the most similar embedding dimension (in terms of Pearson correlation) of a second initialization. Given 20 differently initialized models, we quantify reproducibility of a dimension as the average Pearson correlation between one dimension and its best match across the 19 remaining models. In Table 1, we report the average number of dimensions with a Pearson correlation > 0.8 across the 20 initializations. Selected dimensions are similarly reproducible between VICE and SPoSE (see Table 1). Finally, we investigated whether our uncertainty-based pruning procedure selects reproducible dimensions. We compared the average reproducibility of selected dimensions with the average reproducibility of pruned dimensions, which are discarded by the procedure The average reproducibility of the best subset, i.e., % of dimensions with Pearson’s r > 0.8, is 79.00%, whereas that of the dimensions that were pruned was 0.00%, i.e. our procedure is highly accurate at identifying those dimensions that reproduce reliably.

heading: 3.5 INTERPRETABILITY
text: One of the benefits of SPoSE is the interpretability of the dimensions of its concept embeddings, induced by sparsity and positivity constraints, and empirically tested through experiments in Hebart et al. (2020). VICE constrains the embeddings to be sparse through the spike-and-slab prior, and imposes a non-negativity constraint through applying a rectifier on its latent representations. This
means that, just as in SPoSE, it is easy to sort objects within a VICE dimension by their absolute weight values in descending order, to obtain human judgments of what a dimension represents. In Figure 4, we show the top six objects for four example VICE dimensions of the pruned median model, representing categorical, functional, structural, and visual information. In Appendix A.2 we show the top ten objects for every VICE dimension. Redoing the SPoSE dimension labeling experiments is beyond the scope of this paper, but we provide dimension labels from a small survey in the Appendix.

heading: 4 DISCUSSION
text: In this paper, we introduced VICE, a novel approach for embedding concepts in a non-negative, sparse space, and using those embeddings to predict human behavior in an odd-one-out task. We solve the same problem as an existing method, SPoSE, but using variational inference and a spikeand-slab prior, which is more appropriate for this modeling situation. VICE yields uncertainty information about the solution, enabling a statistical procedure to automatically determine the number of embedding dimensions, as opposed to the data-dependent heuristics that were used in SPoSE. VICE performs as well as SPoSE in terms of accurately predicting human decisions in an odd-one-out task and modeling the probability distribution over those decisions, but using fewer dimensions. However, this is the case only for the large dataset that was originally used to develop SPoSE. VICE performs substantially better than SPoSE on smaller datasets. Moreover, VICE is more stable than SPoSE, as the dimensionality of the embeddings varies less across random initializations. We believe these improvements stem from the combination of the prior and the dimension selection procedure.
We developed VICE with the goal of making it easier to build interpretable embedding spaces to model any type of item, by using an odd-one-out task. We require fewer participants than SPoSE due to higher data efficiency, which makes behavioral experiments more feasible. Our procedure for determining the number of dimensions aims at removing subjectivity, as the scientific motivation often is to discover the minimum number of latent factors required to describe observations. Note that while the user does need to choose the FDR tolerance q, this can be done before looking at any data, based on their degree of conservatism with regards to controlling false discoveries. Hence, it does not cause the problems associated with data-dependent tuning parameters such as regularization parameters or absolute thresholds. Finally, the spike-and-slab prior we use in VICE has intrinsically meaningful hyperparameters, which makes it easier for researchers to specify competing hypotheses about the representation space being studied.

heading: A APPENDIX
text: A.1 EXPERIMENTAL SETUP
Weight initialization We initialized the weights of the encoder for the means of the distributions, Wµ, following a Kaiming He initialization (He et al., 2015). The weights of the encoder for the logarithm of the scales of the distributions, Wlog (σ), were initialized with = − 1sW0µ , such that W 0log(σ) = 1. This initialization allowed us to avoid bias terms within the linear transformations of the encoders, and additionally ensured, through computing σ = exp (log (σ)), that σ is a small continuous number in R+ at the beginning of training.
Hyperparameter grid To find the optimal VICE hyperparameter combination, we performed a grid search over π, σspike, σslab (see Equation 8). The final grid was the Cartesian product of these parameter sets: π = {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}, σspike = {0.125, 0.25, 0.5, 1.0, 2.0}, σslab = {0.25, 0.5, 1.0, 2.0, 4.0, 8.0}, subject to the constraint σspike << σslab, where combinations that did not satisfy the constraint were discarded. We observed that setting σslab > 8.0 led to numerical overflow issues during optimization, which is why 23 was the upper-bound for σslab. For SPoSE, we use the same range as Zheng et al. (2019), with a finer grid of 64 values.
Optimal hyperparameters We found the optimal VICE hyperparameter combination through a two step procedure, for which the validation set was split into equally sized pruning and tuning sets. First, among the final 180 combinations (see Cartesian product above), we applied our pruning method (see Section 2.3.4) to each model and kept the subsets of dimensions that led to the lowest cross-entropy error on one half of the validation set, which we call pruning set. Second, we evaluated each model with pruned parameters on the other half of the validation set, which we refer to as tuning set. We defined the optimal hyperparameter combination as that with the lowest average cross-entropy error on the tuning set across twenty different random initializations. The optimal hyperparameter combinations for VICE were σspike = 0.125, σslab = 1.0, π = 0.4; for SPoSE, it was λ = 5.75.
A.2 OBJECT DIMENSIONS
Here, we display the top 10 objects for each of the 47 VICE dimensions, according to their absolute weight value. As we have done for every other experiment, we used the pruned median model to guarantee the extraction of a representative sample of object dimensions without being over-optimistic with respect to their interpretability (see Section 3.3.1 for how the median model was identified). For each dimension we collected human responses in a small survey with a sample of convenience (n = 9). The labels that are shown below each object dimension represent the most common answer across human responses, when they were asked to name the respective dimension. More than one label is displayed, whenever there was a tie in the most common response. Labels were edited for coherence across similar answers (e.g. "metallic" and "made of metal" were deemed to be the same answer).
While the illustrations for each dimension display the top 10 items, for our survey, in order to avoid biasing our results, we actually show a continuum of items selected from bins centered around the 25, 50, and 75 percentiles in addition to top items, and a random set of items with close to zero weight in the dimension.
METALLIC FOOD
PLANTS ANIMAL
HOME CLOTHES
OUTDOOR WOOD; MADE OF WOOD
POINTY; ELONGATED BODY PARTS
VEHICLE; TRANSPORTATION EXQUISITE; TRADITIONAL
ELECTRONIC COLORFUL
ROUND; CIRCULAR MANY OBJECTS; COLLECTION
STATIONERY; OFFICE SPORTS; GAMES
DECORATIVE; BEAUTIFUL CONTAINER; DRINKS
MARINE; WATER RED
BATHROOM; HYGIENE WAR; WEAPON
BLACK DUST; GRAINY TEXTURE
SPHERICAL; ROUND GREEN
WHITE SKY; FLYING
FLOOR; PATTERN LINES; GRATING PATTERN
MUSIC; SOUND SKY; TALL
INSECTS; PESTS N/A
FIRE; SMOKE FOOT; FOOTWEAR
CHAIN; ROPE; STRAND YELLOW; ORANGE
EYEWEAR; EYES; FACE SPIKY; HAIRY
CYLINDRICAL; ELONGATED STRINGY; FIBROUS
BABY; CHILDREN MEDICAL; HEALTHCARE
ICE; COLD

Review summaries:
This paper presents a technique for learning image embeddings from similarity data provided as odd-one-out judgments over triplets of images (i.e., ball is more similar to apple than car). The authors build on an earlier technique called SPoSE that learns sparse, non-negative embeddings for images by maximizing the probability of choosing the right pair (where this similarity is calculated using the dot-product of image embeddings) with an L1 penalty on embeddings. In this paper, the authors argue that 1) a spike and slab prior is more suited for this setting and 2) a more principled approach to choosing the number of dimensions in learned embeddings is possible.  They present an improved version of SpoSE (called VICE) that uses variational inference to learn embeddings with uncertainties associated with them and place a spike-and-slab prior on the embeddings to encourage sparsity. They then present a way to prune the set of learned dimensions. This calculates the importance of each dimension first (using the learned uncertainties and false discovery rate), then clusters these, and finds the subset of clusters that give the best validation performance.  They argue that their technique is more principled than SpOSE and show that it performs similarly to it in terms of prediction accuracy. However, in small data regime, their model outperforms SpOSE.  

The paper address a problem in cognitive science: identify the embedding of objects in the brain's semantic space based on the subjective judgments of objects' similarity (odd-one-out task). It proposes a new model (VICE) to address a few issues of the previous model (SPoSE). The new method uses a different form of prior (spike-and-slab) for the embedding, variational Bayes with Gaussian posterior, and appear to perform as well as or better than SPoSE in several metrics

The paper proposes a prediction model for the odd-one-out task [Zheng 2019], where the goal is to identify which pair people find to be the most similar within the triplet of pictures. The proposed approach learns a variational model to approximate the distribution over the triplet. The model is compared to SPoSE [Zheng 2019] and shown to outperform on small dataset sizes.

The paper introduces a variational inference method for concept embedding in the odd-one-out task.  The objective is to learn representations that allow to predict the odd object from a triplet.  A variational inference problem is established to learn the representations through a Gaussian variational family with a mixture of two Gaussians as prior (spike-and-slab).  It is not clear how the selection of the variational family fits the prior.  Simultaneously, there is a dimensionality reduction procedure to improve the representations.  The experiments show improvement over the Sparse Positive object Similarity Embedding (SPoSE), and the method is also evaluated with random initialization and it shows to be stable.