VICE: Variational Inference for Concept Embeddings

Abstract: In this paper we introduce Variational Inference for Concept Embeddings (VICE), a novel method for learning object concept embeddings from human behavior in an odd-one-out task. We use variational inference to obtain a sparse, non-negative solution, with uncertainty information about each embedding value. We leverage this information in a statistical procedure for selecting the dimensionality of the model, based on hypothesis-testing over a validation set. VICE performs as well or better than previous methods on a variety of criteria: accuracy of predicting human behavior in an odd-one-out task, calibration to (empirical) human choice probabilities, reproducibility of object representations across different random initializations, and superior performance on small datasets. The latter is particularly important in cognitive science, where data collection is expensive. Finally, VICE yields highly interpretable object representations, allowing humans to describe the characteristics being represented by each latent dimension.

Reviews and author responses:

 Thanks for the detailed response. While the revision certainly improves the quality, I still see the unclear motivation that why improvement over SPoSE is scientifically important to the whole community. If the work is going to prove the effectiveness beyond a single benchmark, I would expect multiple benchmarks. Otherwise, the paper needs to convince in a different way why a solution to the THINGS dataset is sufficient to solve other related issues in the community. In either way, I feel the paper in the current form needs a major revision to constitute a stronger contribution. My final rating remains.

 We thank all of the reviewers for their valuable comments.  In our revision, we addressed several concerns about the clarity and motivation of our method, especially the pruning procedure.  We made the following changes to the paper, which we feel will substantially improve the clarity of our paper.

**Related work**

Reviewer qv3e asked for comparisons to the work by Navarro and Griffiths (2008) and the use of the Indian Buffet Process (Griffiths and Gharamani 2011) to select the number of dimensions.  Reviewer CNX6 asked about comparisons to the work by Roads and Love (2021) and Derby et al. (2018).  We found these related works to be highly relevant to our work, and we now discuss them in the related works section.

**Expanded description of objective function**

In response to reviewer pBzd, who would have benefitted from a more explicit representation of the objective function, we added details in section 2.3.1. "Variational Bayesian Inference" on the following.

- Added equations depicting how the reparameterization trick is used to sample embeddings in a differentiable fashion.

- Rewrote objective function equations to show where the ReLU is applied within the objective function to truncate negative weights to zero.

- Clarified the form of the likelihood (originally defined in Zheng et al. 2019).

- Added details of how minibatches are used to approximate the full data objective function. 

**Clarified prediction targets**


In response to reviewer pBzd, who requested to clarify whether we were predicting one-hot vectors or continuous probabilities, in section 2.3.3 "Predicting the odd-one-out using VICE" we now explicitly state that we consider both types of predictions.

**Rewritten description of pruning**

As reviewers 9iqf, qv3e, and pBzd all asked for more clarification about the pruning method, we significantly expanded section 2.3.4 "Dimensionality Reduction in VICE".  We added the following details.

- Added background information on why dimensions with all objects close to zero occur in SPoSE and VICE, motivating the need for pruning.

- Explained the intuition behind the importance score.

- Described the shortcomings of pruning individual dimensions by importance score, which motivated the clustering approach.

- Described the hypothesized correspondence between importance clusters and feature types (McRae et al. 2005) which informed our choice of the maximum number of clusters.

- In the discussion section, we added a sentence explaining how the threshold q is less data-dependent than the L1 threshold for pruning used in Zheng et al. 2019.

**Space-conserving changes**

- The details on the fully-factorized Gaussian variational distribution, previously contained in their own subsection, were merged into 2.3.1. "Variational Bayesian Inference".
- As the methods section already describes our approach to hyperparameter tuning, we found that the results could still be understood without reference to the specifics of the grid that we employed.  We moved those details of hyperparameter tuning for the experiments to the appendix, and did the same for weight initialization.
- We deleted some of the background material on Bayes' rule, MAP estimators in section 2.2, "SPoSE", and the summation form of the variational inference objective function from section 2.3.1., "Variational Bayesian Inference."

**Changes to appendix**

- Added experimental details that were moved out of the main text.
- Added more details about the survey used to label the dimensions.  Specifically, for each dimension we displayed objects ranging across a continuum of values in a dimension, rather than just the top images.


 Thank you for carefully reading our paper, and for writing a detailed review with many suggestions and questions, and for the additional references and comparisons that would strengthen our paper.  In the following, we reply to some of your specific questions.

**If the embeddings are non-negative (due to ReLU), why not use a prior that also makes this assumption (rather than a gaussian)?**

We performed variational inference on the embedding weights, which can be positive or negative, before applying the ReLU. As Reviewer 9iqf says, an alternative method is to perform variational inference on the weights after applying the ReLU and using a prior that is constrained to positive real numbers. However, the fact that the ReLU function is non-invertible makes it challenging to easily evaluate the probability density function of the variational distribution created by applying a ReLU to a Gaussian. This is needed to approximate the KLD with the methods that we used. If we replaced the ReLU with the absolute value function, we could use a folded Gaussian variational distribution for the embedding. However, finding the mode of a folded Gaussian is cumbersome. This is not ideal, since using the mode of the embedding distribution is extremely helpful in downstream applications that use a single embedding vector for each item. Using other variational distributions for the embeddings that are defined only for positive values, such a log-normal distribution, is something that could be investigated in future work.

**And why would you prune by cluster? If a cluster has features that are similar to each other, wouldn't you want to keep at least some from each cluster and prune the rest (because they are redundant)?**

The point of clustering dimension importance scores is to identify entire groups of dimensions (rather than individual dimensions) which are too noisy to be interpretable.  We expect such groups of dimensions to exist because of the existence of very sparse features that could drive similarity judgments (McRae et al. 2005).  Furthermore, as McRae et al. 2005 discusses, these features can be grouped into different feature types, such as categorical, functional, encyclopedic, visual-perceptual and non-visual-perceptual.  We further hypothesized that features with the same feature type have similar sparsity and magnitude profiles, and hence would have importance scores that cluster together.  As such, these clusters are not based on feature similarity (so the features within a cluster are not redundant), but rather based on similarity by importance.  We prune by an entire cluster because if a given feature is uninformative (e.g. too noisy), then features with similar importance scores are likely to be similarly uninformative.
 
**For example, are you clustering a single score per dimension, or large vectors of importance scores (one for each sample) per dimension?**

Each dimension is assigned a single importance score.  The intuition behind the importance score is as follows.  A dimension where no objects have posteriors informatively differ from the prior is completely uninformative and hence should be assigned an importance of 0. There may also exist dimensions where the number of objects with informative posteriors is greater than 0, but still too small to merit inclusion in the final embedding.  Hence, we end up calculating importance as the number of objects that plausibly have non-zero weights as controlled by the False Discovery Rate.

**Why cluster the dimension importance scores and assume a max of 6 dimensions?**

We set a maximum limit on the number of clusters to prevent the clustering procedure from finding a degenerate solution with an excessive number of singleton clusters (Archambeau et al. 2003).  To decide the value of the limit, we made an educated guess on the maximum number of clusters in data based on the assumption that the existence of importance clusters is due to the fact that there exist qualitatively distinct "feature types" such as categorical, functional, encyclopedia, visual-perceptual, and non-visual perceptual. (McRae et al. 2005).  Examining the embeddings in Zheng et al. 2019, we have seen that larger dimensions (sorted by total L1 norm) tend to be categorical, while intermediate dimensions tend to be functional, and smaller dimensions correspond to visual-perceptual properties--hence, we know there could plausibly be at least three different importance clusters corresponding to those three feature groups of categorical, functional, and visual-perceptual.  Given that there could be even more categories than these, we allow for double that amount and set the maximum to 6. 



 **The pruning procedure is hard to follow**

Other reviewers also noted this, so we will clarify the pruning and its motivations in the revision.

**An ablation study is recommended.**

We agree, but regrettably we are unable to add a thorough ablation study in the limited time before the end of the author response period.

Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2017). Variational inference: A review for statisticians. JASA, 112(518), 859-877.

Agarwal, S., Wills, J., Cayton, L., Lanckriet, G., Kriegman, D., & Belongie, S. (2007, March). Generalized non-metric multidimensional scaling. In Artificial Intelligence and Statistics (pp. 11-18). PMLR.

Robbins, H. and Monro, S. A stochastic approximation method. Annals of Mathematical Statistics, 22(3):400– 407, 1951.

Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

Donoho, D. L., & Stodden, V. C. (2004). When does non-negative matrix factorization give a correct decomposition into parts?.

 We thank you for reading our paper and raising many detailed questions which serve to help us clarify our paper.  Please see our responses to some of your questions, and please look forward to our rebuttal response, to be posted in a few days, that will further address the issues that you raised.

**Why not use a mixture for the variational family as well?**

Many of the use cases of embeddings rely on having a single embedding vector for each item. This led us to use a unimodal variational distribution, which allowed us to use the mode of the variational distribution, in this case the mean of a Gaussian, as a representative embedding for an item. More complex variational families may explain the data better, but may not be as useful for downstream applications, such as using the embeddings as regression targets. 

**What is the model?**

VICE is a probability model for triplet choice probabilities, parametrized by a matrix of embedding weights. It can be considered as a variational Bayesian (Blei 2017) method for learning a non-Euclidean multidimensional scaling for triplet choice data (Agarwal 2007).  The probability model itself is the same as that of Zheng et al. 2019.

**Is an encoder used?**

VICE does not use an encoder that takes images as inputs.  VICE explains behavior of human subjects who see the images and choose the odd-one-out.  We do not assume that humans are limited to only using visual information.  Hence, rather than attempting to model any transformation from images to feature space, we learn the features for each item directly based on behavior.  This way, we can in principle learn features that rely on non-visual information.

**What is the training objective?  How is the training done?**

The objective function is written in equation 8, taking M=1 for the number of samples.  We use a variant of stochastic minibatch gradient descent (Robbins and Monro, 1951), Adam (Kingma and Ba, 2014).  Each iteration computes a gradient over a minibatch of triplets.  The sampled embedding vectors drawn from equation 10 are used to compute similarities according to equation 1 which are used to compute the probability of each of the odd-one-out choices in equation 2.  Subtracting these log probabilities from the KL penalty gives the objective function for the minibatch.  The gradient of the minibatch is used to compute the update using the Adam.  We will write out the steps for computing the updates explicitly in the revision.

**Do you use the probabilities from softmax (eq 2) or do you predict a one-hot vector?**

The model can either be used to predict the triplet choice, in which case we report the pair with highest predicted probability, or we can predict the probability vector for all the choices.  We will clarify in the revision that we consider both types of predictions.

**Why zero-truncate the weights below 0?**

The conceptual foundation of the model we are using is built on the assumption of non-negative similarities.  Allowing negative weights introduces the possibility of potentially having negative similarities.  The non-negativity constraint we adopt is motivated by the fact, observed in many different applications, that a non-negativity constraint on parameters often greatly improves the sample complexity of recovering sparse parameters (Donoho and Stodden 2004).  This is in addition to the improvements to interpretability that we qualitatively observe in non-negatively constrained embeddings.

**Couldn’t looking at top-scoring items bias the interpretation?**

The thought also occurred to us that looking only at top items might be misleading, so in our survey we actually show a continuum of items selected from bins centered around the 25, 50, and 75 percentiles in addition to top items, and a random set of items with close to zero weight in the dimension.  In the paper, we only show the top 6 items for the sake of readability and to keep the file size of the pdf manageable.  We will clarify this in the revision.




 **“L Vilnis et al, Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures, ACL 2018
 O Ganea et al, Hyperbolic Entailment Cones for Learning Hierarchical Embeddings, ICML 2018”**

The methods in these papers -- and this class of methods in general -- attempt to solve a similar problem to SPoSE and VICE, to the extent that they provide the ability to turn relations between items into an embedding (and, in particular, model hierarchy). They both run experiments on WordNet, where those relations were described by human annotators. These relations could be taken as somewhat analogous to behavior, but the former are far cleaner in our experience. This said, these methods are not meant to produce interpretable representations. It is also not clear that they would work with random samplings of relationships between items akin to those in the SPoSE paper and our work. We thank the reviewer for pointing us to these methods, as they might be worth trying on our type of data in future work. However, they are well beyond the scope of our paper, which is to make an approach like SPoSE both more principled and more data efficient.

Olshausen, B. A., & Millman, K. J. (1999, January). Learning sparse codes with a mixture-of-Gaussians prior. In NIPS (Vol. 12, pp. 841-847).

Lam, K. C., Pereira, F., Vaziri-Pashkam, M., Woodard, K., & McMahon, E. (2021). Mental representations of objects reflect the ways in which we interact with them. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 43, No. 43).

Ahn, S., Zelinsky, G., & Lupyan, G. (2019). Effects of Linguistic Labels on Learned Visual Representations in Convolutional Neural Networks: Labels matter!.
https://openreview.net/pdf?id=r1xH5xHYwH


 We thank you for reading our paper and for suggesting many related works that would indeed allow us to better contextualize our contribution.  Please see our responses below.

**“Even though the results indicate successful in improving in smaller datasets (Fig 3), I do not understand how significant the result is without any generalization verification.”**


With regards to the generalization, note that we developed the method using the same datasets as [Zheng 2019], partitioned into train, validation, and test. Only when we had the final method did we apply it to the completely held-out test dataset, where we report results on both accuracy and KL-divergence to human performance. We most definitely would like to try our method on additional datasets, but this was beyond what was feasible in either time before submission or space in it.

**“It might be the case that there is a scientific value in improving the benchmark performance of the odd-one-out dataset [Zheng 2019], but as a reviewer, I do not have any background in judging the statement in Sec 1 saying “growing interest by cognitive scientists using SPoSE”.”**

We are judging this by the number of citations of [Zheng 2019] or [Hebart 2020] and, specifically, those that use SPoSE as a baseline method (e.g. [Ahn et al. 2019]) or as a prediction target (e.g. [Lam et al. 2021]). 

**“Although I do not deeply understand the cognitive science context, I feel this paper misses the formal discussion in terms of representation learning in the natural language processing community. Even the simplest comparison done in [Zheng 2019] to NLP baselines (synset or NNSE) are missing in this work.”**

We considered including NLP baselines -- both interpretable and dense vectors, as in NNSE and synset vectors -- respectively, but ultimately decided against it. The reason is that the dimensions we identify are very similar to those of SPoSE, quantitatively, and so are the interpretations human subjects give to them. Given that the comparison was already done in [Zheng 2019], with favourable results for SPoSE, we felt it would have added very little information here.

**Roads and Love (2021) propose a related method.**

We thank the reviewer for bringing this paper to our attention.The method proposed by Roads and Love (2021) is an embedding model for predicting human behaviour for an 8-rank-2 behavioural task. They use variational inference to learn a distribution of embeddings, in order to use the uncertainty about the embedding of each item to improve the efficiency of stimuli selection using active learning. Specifically, they use a fully factorized Gaussian variational distribution and prior. The embedding is not built to be interpretable. VICE seeks to have interpretable embeddings by making the embeddings sparse and positive. A mixture-of-Gaussians prior can induce sparsity (Olshausen and Millman, 1999), unlike a fully factorized Gaussian prior. In the area of cognitive science, having an interpretable embedding is very important, making the embedding learned by VICE more desirable than the embedding learned by the model proposed by Roads and Love (2021).


**“S Derby et al, Feature2Vec: Distributional semantic modelling of human property knowledge, EMNLP 2019”**

We thank the reviewer for pointing us to the Feature2Vec paper, which we were not aware of and will add to the related work section. In this case, the authors are learning a mapping from semantic property norms (binary, sparse, several thousand for several hundred objects) to a dense word embedding space (GloVe), using a neural network. This is then used to predict embedding vectors for combinations of properties. The internal representation of this network could potentially be used to extract higher-level representations of objects that combined multiple properties (as [Zheng 2019] shows SPoSE does), but this was not the goal of the paper. 

**“A Laverghetta et al, Can Transformer Language Models Predict Psychometric Properties?, STARSEM2021”**

This paper looks at the feasibility of using neural language models (183 transformer, 57 LSTM) to predict the answers to a subset of the GLUE diagnostic questions, benchmarked against human answers and thus deemed psychometric properties. The internal representations in these language models are dense, thus not directly interpretable, and our goal is not to predict semantic judgments in a supervised manner (as they do).




 **Comparison to other approaches?**

We thank the reviewer for the reference to Navarro and Griffiths, which uses the Indian Buffet Process.  We will add this reference to our related works section.  Their similarity model is very similar to our own, with three key differences.  (1) They use binary features, rather than continuous features.  (2) They work with continuous-valued similarity ratings directly, whereas in our model we infer similarities from triplet comparisons.  (3) They use IBP to select the number of dimensions, while we start with a large number of dimensions and then use pruning to determine the number of features.  It would be interesting to try binary features, as well as using IBP to determine the number of features, but the Bayesian inference used in their work, which relies on Markov Chain Monte Carlo, may not scale well to the number of items and observations in the Zheng 2019 et al. dataset.

Graves, A. (2011). Practical variational inference for neural networks. Advances in neural information processing systems, 24.

Verhoeven, K.J., Simonsen, K.L. and McIntyre, L.M. (2005), Implementing false discovery rate control: increasing your power. Oikos, 108: 643-647. https://doi.org/10.1111/j.0030-1299.2005.13727.x


 We thank you for your detailed reading of our paper, and raising many key questions that will enable us to make the paper clearer.  Please see below our responses to your questions.



**How were p-values for pruning computed? Why use p-values in this manner?**

Our approach to calculate p-values based on the variational distribution is adapted from a heuristic recommended by the Graves 2011 paper, "Practical variational inference for neural networks."  The idea is to use the Bayesian posterior probability of the weight being zero as an indicator of its importance--if a weight is zero with a large probability, then it is not important.  The p-value is calculated as the probability that the weight is truncated, based on the Gaussian tail probability below zero of the variational distribution. Truncation of the weights at 0 is not part of the pruning, but part of the method itself, which assumes non-negative weights. We will describe the whole pruning approach in greater detail and clarity in the revision.
 

**In what way are weights with low p-values "predictive"?**

We define a weight as "predictive" if it increases the probability of observed choices. If it does so in the training data, then the weight lowers the negative log-likelihood in the objective.  To connect this to the p-value, note that weights with low p-value necessarily have high penalties in the KL divergence (because low p-values implies either high mean or low variance or both, and this leads to high KL divergence from the zero-centered prior).  In order to offset the cost within the KL penalty, these particular weights must contribute substantially to lowering the negative log-likelihood in the objective function.  In short, a low p-value implies that the weight contributes greatly to lowering the negative log-likelihood, indicating that the weight is predictive.


**It seems strange for dimensions to have mean close to zero for all items.**

For interpretability purposes, it is crucial that the model does not use more dimensions than necessary.  Zheng et al. (2019) accomplished this through the sparsity-inducing penalty that causes dimensions to shrink towards zero if they do not contribute to explaining the data. Here, the KL penalty we use has a similar effect of causing uninformative weights to shrink.  Note however that these uninformative weights do not totally go to zero because of noise in the gradients (both for VICE and SPoSE). Hence, we eliminate the unused dimensions by pruning.


**Isn't the q threshold for the FDR procedure as arbitrary as the threshold for SPoSE pruning?**

While q is also a tuning parameter, consider that not all tuning parameters are equally problematic.  The threshold for pruning used by SPoSE is especially problematic because (1) there is little guidance on how to choose it, (2) it could vary greatly depending on the dataset and (3) even on the optimizer and optimization parameters used. Whereas the q threshold benefits from a large literature in statistics on the meaning of significance and error control thresholds (Verhoeven et al. 2005), and is intuitively understandable (as the fraction of false discoveries that would be tolerable), and can arguably be set independently of the properties of the data itself.  Indeed, we recommend choosing q ahead of time based on the level of conservativeness desired, before looking at the data.


**Why cluster the dimension importance scores and assume a max of 6 dimensions?**

The point of clustering dimension importance scores is to identify entire groups of dimensions (rather than individual dimensions) which are too noisy to be interpretable.  We expect such groups of dimensions to exist because of the existence of very sparse features that could drive similarity judgments (McRae et al. 2005).  Furthermore, as McRae et al. 2005 discusses, these features can be grouped into different feature types, such as categorical, functional, encyclopedic, visual-perceptual and non-visual-perceptual.
We set a maximum limit on the number of clusters to prevent the clustering procedure from finding a degenerate solution with an excessive number of singleton clusters (Archambeau et al. 2003).  Examining the embeddings in Zheng et al. 2019, we have seen that larger dimensions (sorted by total L1 norm) tend to be categorical, while intermediate dimensions tend to be functional, and smaller dimensions correspond to visual-perceptual properties--hence, we know there could plausibly be at least three different importance clusters corresponding to those three feature groups of categorical, functional, and visual-perceptual.  Given that there could be even more categories than these, we allow for double that amount and set the maximum to 6. 




 **And finally, how about just sorting the dimensions by importance and taking the top N? How well does this work compared to the clustering strategy?**
Taking the top N could be quite reasonable depending on how N is decided.  In fact, while not shown in the paper, during the development process, we tried choosing the number of dimensions based on cross-validation performance.  This led to high variance in the number of dimensions, which is why we started to develop a clustering-based approach.  We found that the clustering approach presented in the paper does decrease the variance of the number of dimensions for the THINGS dataset.

**I think more details on this in the paper would be useful**
We agree, and with the extra page allowed in the revision we will clarify the description of the pruning method and its motivation.

Archambeau, C., Lee, J. A., & Verleysen, M. (2003, April). On Convergence Problems of the EM Algorithm for Finite Gaussian Mixtures. In ESANN (Vol. 3, pp. 99-106).
McRae, K., Cree, G. S., Seidenberg, M. S., & McNorgan, C. (2005). Semantic feature production norms for a large set of living and nonliving things. Behavior research methods, 37(4), 547-559.

This paper presents a technique for learning image embeddings from similarity data provided as odd-one-out judgments over triplets of images (i.e., ball is more similar to apple than car). The authors build on an earlier technique called SPoSE that learns sparse, non-negative embeddings for images by maximizing the probability of choosing the right pair (where this similarity is calculated using the dot-product of image embeddings) with an L1 penalty on embeddings. In this paper, the authors argue that 1) a spike and slab prior is more suited for this setting and 2) a more principled approach to choosing the number of dimensions in learned embeddings is possible.

They present an improved version of SpoSE (called VICE) that uses variational inference to learn embeddings with uncertainties associated with them and place a spike-and-slab prior on the embeddings to encourage sparsity. They then present a way to prune the set of learned dimensions. This calculates the importance of each dimension first (using the learned uncertainties and false discovery rate), then clusters these, and finds the subset of clusters that give the best validation performance.

They argue that their technique is more principled than SpOSE and show that it performs similarly to it in terms of prediction accuracy. However, in small data regime, their model outperforms SpOSE.

 Overall, I think it is an interesting paper. I like the spike-and-slab prior and the variational treatment. I agree with the authors that these are better suited to this setting.

My main concern with the paper is that the motivation is not really clear. What would these embeddings be used for after learning? It looks like the authors have some applications in cognitive science in mind but I wasn't sure what these are exactly. How will these embeddings be useful in cognitive science experiments? I think a case study of where these embeddings were used would add significantly to the paper. Also, it would be nice to motivate this technique for the larger ICLR audience as well (not only cognitive scientists).

My other concern with the paper is that it's not very clear if the contributions are significant enough. 

VICE performs similarly to SpOSE in general, only except it outperforms SpOSE on smaller datasets. However, it wasn't clear to me how important this is. Mainly because the smallest dataset in the experiments, which has 67.500 triplets, doesn't seem very small to me from a psychology experiment perspective. (Assuming an experiment with 100 subjects, which is pretty large, perhaps you could collect 10.000 triplets). It would be very valuable to see the performance of VICE on even smaller datasets.

I agree the pruning strategy is more principled than SpOSE's but it still seemed unusual to me in a couple of respects. First, it wasn't clear to me why you would cluster the dimension importance scores (and assume a max of 6 dimensions). I think more details on this in the paper would be useful. For example, are you clustering a single score per dimension, or large vectors of importance scores (one for each sample) per dimension? And why would you prune by cluster? If a cluster has features that are similar to each other, wouldn't you want to keep at least some from each cluster and prune the rest (because they are redundant)? And finally, how about just sorting the dimensions by importance and taking the top N? How well does this work compared to the clustering strategy?

Finally, I'd also have liked to see comparisons to other techniques. I understand that the technique is built on SpOSE but there are many techniques that would be applicable to this problem. One class of methods that are especially suited to this setting are non-parametric Bayesian models (like Indian Buffet Process)[1] that can determine the optimal number of dimensions (without any explicit pruning strategy). One can also use a large multi-modal model like CLIP (OpenAI) to get embeddings for images and it might be interesting to see how well these do. Or if the authors think these models are not applicable, then a brief discussion of why they aren't would be valuable I think.

Below are some other minor points

- In 3.5, the authors mention that VICE embeddings are passed through a rectifier to make them non-negative. This should be mentioned earlier in the paper. Also, if the embeddings are non-negative, why not use a prior that also makes this assumption (rather than a gaussian)?

- In 3.2, they mention variances are initialized to be small. Why? Generally, people initialize these to be close to 1.

- A minor organization suggestion. At the end of page 6, they mention separate pruning and tuning validation sets. It would be nice to mention this earlier (when talking about pruning strategy for example).

[1] Navarro DJ, Griffiths TL (2008) Latent features in similarity judgments: A nonparametric Bayesian approach. Neural Computation.

 Overall, I think the contributions of this paper are unfortunately not strong enough for a clear acceptance. The paper would benefit from better motivation of the technique, more extensive experimental evaluation, and comparison to other existing models.

The paper address a problem in cognitive science: identify the embedding of objects in the brain's semantic space based on the subjective judgments of objects' similarity (odd-one-out task). It proposes a new model (VICE) to address a few issues of the previous model (SPoSE).
The new method uses a different form of prior (spike-and-slab) for the embedding, variational Bayes with Gaussian posterior, and appear to perform as well as or better than SPoSE in several metrics The topic addressed by the paper is an interesting and important one for cognitive science.
It is nice that the paper pointed out a few limitations of the previous model, for example, the L1 penalization may not match the actual distribution. The experiments generally appear solid. The better performance with small training dataset as shown in Fig 3 is quite attractive.

But I also find the performance to be comparable to SPoSE in most cases and only better in a few aspects.

Other major issues:
I did not really get how 2.3.5.1 was done. Since you mentioned mu and sigma, I suppose they refer to the approximate posterior distribution in the variational bayes. But I did not get how you actually truncate it. where is the truncated Gaussian used? Is it only used for calculating p-value? Because this p-value has different interpretation than the one typically used in classical hypothesis testing, I recommend  you make a bit more explanation of what probability the p actually indicate.
I think such hypothesis testing require the samples to be independent from each other. But because of the prior introduced, the posterior mean is biased. Does it still make sense to use such a p-value to assess non-zero embedding mean? Also, what does it mean to be "predictive" in this paragraph? 

It seems also conceptually strange for any items with mean close to zero in all dimensions. What does the representation mean, if the p-value is high for such item over all dimensions? It sounds like the object would be treated as "noise"?

It was criticized that the criterion for determining how many dimensions the solution should have in SPoSE is heuristic and subjective, but I find that allowing users to choose q value in FDR to determine dimension importance also subjective. 

2.3.5.3 and 2.3.5.4: The choice of pruning by cluster sounds quite ad hoc. The purpose is only for reducing variance. But is there any theoretical rationale to justify this? GMM assumes each component is a latent cause. But why would dimensions of similar importance to be of the same latent cause and share the same fate of being either retained or pruned? 



 I find the work generally interesting. But the advantage is not super apparent. It seems to mainly help when the training data is small.
I find the detailed procedure a bit ad hoc. The phrasing of the procedure seems a bit confusing when it uses terms of frequentist statistics, but seem to not mean their original meanings.

The paper proposes a prediction model for the odd-one-out task [Zheng 2019], where the goal is to identify which pair people find to be the most similar within the triplet of pictures. The proposed approach learns a variational model to approximate the distribution over the triplet. The model is compared to SPoSE [Zheng 2019] and shown to outperform on small dataset sizes. ## Strengths

The paper clearly describes the main idea of applying variational approach to the concerned odd-one-out task. Compared to SPoSE, the proposed approach introduces a stochastic process to explicitly consider uncertainty, and switch the prior distribution to Gaussian mixture for better fitting to the data. The motivation seems clear and the approach looks reasonable.

## Weaknesses

The paper has limited significance due to the narrow focus on improvement to SPoSE [Zheng 2019]. The paper seems to only concern the benchmark performance in the single dataset of [Zheng 2019], which might be simply overfitting the model to a specific dataset. Even though the results indicate successful in improving in smaller datasets (Fig 3), I do not understand how significant the result is without any generalization verification. In this sense, the paper fails to convey whether if the proposed approach has significant technical contribution.

There seems a recent work on proposing alternative benchmark.

- B Roads and B Love, Enriching ImageNet With Human Similarity Judgments and Psychological Embeddings, CVPR 2021

It might be the case that there is a scientific value in improving the benchmark performance of the odd-one-out dataset [Zheng 2019], but as a reviewer, I do not have any background in judging the statement in Sec 1 saying “growing interest by cognitive scientists using SPoSE”.

On learning embedding representations, there are different attempts other than Gaussian.

- L Vilnis et al, Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures, ACL 2018
- O Ganea et al, Hyperbolic Entailment Cones for Learning Hierarchical Embeddings, ICML 2018

Although I do not deeply understand the cognitive science context, I feel this paper misses the formal discussion in terms of representation learning in the natural language processing community. Even the simplest comparison done in [Zheng 2019] to NLP baselines (synset or NNSE) are missing in this work.

There seems somewhat recent relevant work.

- A Laverghetta et al, Can Transformer Language Models Predict Psychometric Properties?, STARSEM2021
- S Derby et al, Feature2Vec: Distributional semantic modelling of human property knowledge, EMNLP 2019 The paper has a clear motivation to learn better concept representation than SPoSE [Zheng 2019], but that seems to result in the insignificant focus on the improvement to SPoSE at small data sizes. I feel that this work should better contextualize the scientific motivation among language literature, and consider generalization over the THINGS dataset. My initial rating is below the acceptance threshold.

The paper introduces a variational inference method for concept embedding in the odd-one-out task.  The objective is to learn representations that allow to predict the odd object from a triplet.  A variational inference problem is established to learn the representations through a Gaussian variational family with a mixture of two Gaussians as prior (spike-and-slab).  It is not clear how the selection of the variational family fits the prior.  Simultaneously, there is a dimensionality reduction procedure to improve the representations.  The experiments show improvement over the Sparse Positive object Similarity Embedding (SPoSE), and the method is also evaluated with random initialization and it shows to be stable. Strengths:
- The paper continues an interesting problem (odd-one-out) classification.
- The idea of using variational inference to understand the problem is straightforward and interesting.

Weaknesses:

- In (8), it is not clear whether $p(\mathcal{D}_{\text{train}} \mid X)$ is the same normalized similarity used in SPoSE (2).

- What is the impact of using a Gaussian variational family and having a mixture of Gaussians as a prior?  My guess is that your variational distribution $q$ will end up wide and will try to encompass the mixture, or if the spike is too dominant it may collapse to it.  Why not use a mixture for the variational family as well?

- What is the model you are training?  What is the definition of your encoder that produces the embeddings $X$?  Does it receives a single image or a triplet?

- What is the training objective that you are using?  And what are the pieces of your model?  It is not clear whether you are training an encoder $q$ that receives a single data point and minimizes a batch of triplets w.r.t. the average of the normalized similarities (8), or if you are also using a prediction head (or classifier) for the odd-one-out task as well (12) to regularize the model.

  In other words, are you training your encoder with only the similarities from the triplets, or you are using the odd-one-out classification task as well?

- In Section 2.3.4, the prediction of the odd-one-out seems to be the normalized similarity pair-wise.  So, the reader will assume that the prediction is through doing the 3 pairs from the triplet and selecting the one with minimum similarity.  However, in Section 3.3.1, it seems that the prediction occurs as a one-hot-vector prediction.  Which one is it?  The usage of your model for the prediction task must be clearly stated and defined.

- Why is the truncation of the dimensions needed?  One could still traverse the dimensions and see clusters of points through them. 

- Similar to the previous comment, the interpretability of the dimensions explores the maximum values to check for clusters in this neighborhood.  However, wouldn't it be interesting to see the traversals through the dimension to see how the samples change or not?  

  Isn't it bias to look only at the clump of samples from the high values in the dimension (assuming that is the weight)?  One will expect to see similar clusters since the embeddings are close together for the maximum values. Similar to other parameters, why not to do an ablation study over the truncation too?

- I found hard to follow the process of doing the dimensionality reduction.  It seems to be summarized, and I think its incorporation to the model impacts the final performance.  I suggest to improve the description and details of how this process works.  If space is needed, instead of reproducing SPoSE (Section 2.2), consider removing it.

- How one can know whether the improvement of the representations come from the variational representation or from the different steps within the dimensionality reduction process?  I would recommend to do an ablation study to understand the contribution of the different steps on the dimensionality reduction to understand their impact on the final representations. The proposal extends SPoSE within a variational framework.  The proposal uses a Gaussian variational family with a mixture of Gaussians prior (the limitations of this selection or its impact is not discussed).  The paper defines a normalized similarity within triplets as a regularization on top of the KL-divergence for the variational inference problem.  The model and the training objective are not clearly defined.

The results show improvement over the SPoSE baseline.  However, the different components and parts of the proposal are not validated.  Thus, it is hard to understand where the benefit is coming from (e.g., either the variational model, or the different post-processing steps for dimensionality reduction).  The interpretabiltiy, while interesting, seems biased and the way the dimensions were selected may have influenced the observed results. 

Overall, the method's presentation needs improvement.  In particular, the definition of the networks used and the objectives used to train it (i.e., the explicit loss functions).  The different components of the proposal should be validated and an extensive ablation study is needed to understand where the improvements are coming from. 

Meta-review: The authors propose Variational Inference for Concept Embeddings (VICE), a method to learn representations such that an odd object can be detected given a triplet (i.e. the odd-one-out task). The authors build on Sparse Positive object Similarity Embedding (SPoSE) which learns sparse, non-negative embeddings for images by placing a zero-mean Laplace prior. Claimed contributions include replacing it with a spike-and-slab Gaussian mixture prior, and a principled approach to choosing the subset of the dimensions of the learned embeddings. The empirical results show improvements over the SPoSE baseline.

The reviewers appreciated the empirical improvements over SPoSE and accept that a more informative prior might lead to improved results. However, the **motivation, novelty and significance** of the proposed method doesn’t meet the acceptance criteria for ICLR. After the rebuttal and the discussion phase the reviewers felt that the work necessitates a major revision (notwithstanding the remaining issue with limited novelty), and raised the following as the main improvement points: 
- Clarifying the motivation and significance.
- Stronger empirical validation and generalization beyond the THINGS dataset.
- Address the discrepancy with analyzing GMM priors, but using unimodal Gaussians in the implementation.
- Comparing the chosen prior to other prior distributions and justifying the design choices.

Decision: Reject