VICE: Variational Interpretable Concept Embeddings
Authors: Lukas Muttenthaler
Charles Yang Zheng
Patrick McClure
Robert A. Vandermeulen
Martin N Hebart
Francisco Pereira
Venue: NeurIPS 2022 Accept (2022)
Keywords: Cognitive Science
Latent Variable Models
Variational Inference
Concept Representations

Abstract: A central goal in the cognitive sciences is the development of numerical models for mental representations of object concepts. This paper introduces Variational Interpretable Concept Embeddings (VICE), an approximate Bayesian method for embedding object concepts in a vector space using data collected from humans in a triplet odd-one-out task. VICE uses variational inference to obtain sparse, non-negative representations of object concepts with uncertainty estimates for the embedding values. These estimates are used to automatically select the dimensions that best explain the data. We derive a PAC learning bound for VICE that can be used to estimate generalization performance or determine a sufficient sample size for experimental design. VICE rivals or outperforms its predecessor, SPoSE, at predicting human behavior in the triplet odd-one-out task. Furthermore, VICE's object representations are more reproducible and consistent across random initializations, highlighting the unique advantage of using VICE for deriving interpretable embeddings from human behavior.

VICE: Variational Interpretable Concept Embeddings
Lukas Muttenthaler'
Charles Y. Zheng
Machine Learning Group Technische Universität Berlin BIFOLD Berlin, Germany
Machine Learning Team, FMRI Facility National Institute of Mental Health Bethesda, MD, USA
Patrick McClure Department of Computer Science Naval Postgraduate School Monterey, CA, USA
Robert A. Vandermeulen
Machine Learning Group
Technische Universität Berlin
BIFOLD Berlin, Germany
Martin N. Hebart
Francisco Pereira
Vision and Computational Cognition Group MPI for Human Cognitive and Brain Sciences Leipzig, Germany
Machine Learning Team, FMRI Facility National Institute of Mental Health Bethesda, MD, USA
Abstract
A central goal in the cognitive sciences is the development of numerical models for mental representations of object concepts. This paper introduces Variational Interpretable Concept Embeddings (VICE), approximate Bayesian method for embedding object concepts in a vector space using data collected from humans in triplet odd-one-out task. VICE uses variational inference obtain sparse, non-negative representations of object concepts with uncertainty estimates for the embedding values. These estimates used to automatically select the dimensions that best explain the data. We derive a PAC learning bound for VICE that can be used to estimate generalization performance or determine sufficient sample size for experimental design. VICE rivals or outperforms its predecessor, SPoSE, at predicting human behavior in the triplet odd-one-out task. Furthermore, VICE's object representations are more reproducible and consistent across random initial- izations, highlighting the unique advantage of using VICE for deriving interpretable embeddings from human behavior.
Introduction
Human knowledge about object concepts encompasses many types of information, including function or purpose, visual appearance, encyclopedic facts, taxonomic characteristics. A central question in cognitive science concerns the representation of this knowledge and its use across different tasks. One approach to this question is inductive and lets subjects list properties for hundreds to thousands of objects [28] 10 20], yielding large lists of responses about different types of properties. Specifically, objects are represented vectors of binary properties. While this approach is agnostic to downstream prediction tasks, it may be biased since subjects may leave out important features and/or
affiliated with Max Planck Institute Human Cognitive and Brain Sciences, Leipzig, Germany. Berlin Institute for the Foundations of Learning and Data, Germany.
I Work was partially done while affiliated with the National Institute of Mental Health, Bethesda, MD, USA.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

mention unimportant ones. For example, one may forego a general property (e.g., "is an animal") while providing highly specific fact (e.g., "is found in Florida"). In an alternative, deductive approach, researchers postulate dimensions of interest and subsequently let subjects rate objects in each dimension, Binder et al. 33] employed such an approach and collected ratings for hundreds of objects, verbs, and adjectives. These ratings were gathered over 65 dimensions, reflecting sensory, motor, spatial, temporal, affective, social, and cognitive experiences. Nonetheless, is desirable to discover object representations that are not biased by the conducted behavioral task and whose dimensions are interpretable without necessitating a priori assumptions about their semantic content.
Recently, Zheng et al. [46] and Hebart et al. [9] introduced SPoSE, model of the mental representa- tions of 1,854 objects in a 49 -i-dieeniinall space. The model was learned from human judgments about object similarity, where subjects were asked to determine an odd-one-out object in random triplets of objects. SPoSE embedded each object in a vector space so that each dimension non-negative and sparse (most objects have a close-to-zero entry for given dimension). The authors showed that the embedding dimensions of objects were interpretable and that subjects could coherently label what the dimensions were "about," ranging from categorical (e.g., animate, food) to functional (e.g., tool), structural (e.g., made of metal or wood), or visual (e.g., coarse pattern). The authors hypothesized that interpretability arose from combining positivity and sparsity constraints so that no object was represented by every dimension, and most dimensions were present for only a few objects. In addition, SPoSE could predict human judgements close to the estimated best attainable performance [46] 19).
Despite its notable performance, SPoSE has several limitations. The first stems from the use of an li sparsity penalty to promote interpretability. In SPoSE, 6 to 11 dominant dimensions for an object account for most of the prediction performance. These dimensions are different between objects. A potential issue with enforcing SPoSE to have even fewer dimensions is that it may cause excessive shrinkage of the dominant values [.. Second, when inspecting the distributions of values across objects, most SPoSE dimensions do not reflect the exponential prior induced by the é penalty, Overcoming this prior may lead to suboptimal performance and inconsistent solutions, specifically in low data regimes. Third, SPoSE uses an ad-hoc criterion for determining the dimensionality of the solution via an arbitrary threshold on the density of each dimension. Finally, SPoSE has no criterion for determining convergence of its optimization process, nor does it provide any formal guarantees on the sample size needed to learn a model of desired complexity.
To overcome these limitations we introduce VICE, variational inference ( (VI) method for embedding object concepts with interpretable, sparse, and non-negative dimensions. We start by discussing related work and description of the triplet task and SPoSE, followed by our presentation of theory and experimental results.
Contribution 1: VICE solves major limitations of SPoSE First, VICE encourages shrinkage while allowing for small entry values by using spike-and and- prior 12 15 deem this more appropriate than an exponential prior, because importance the value an object takes in dimension - is different from relevance whether the dimension is applicable to that object and both can be controlled separately. Second, we use VI with a unimodal posterior for representing each object in a dimension which yields mean value and an uncertainty estimate. While unimodality makes it possible to use the mean values as representative object embeddings, the uncertainty estimates allow us to use a statistical procedure to automatically select the dimensions that best explain the data. Third, we use this procedure to introduce convergence criterion that reliably identifies representational stability, i.e. the consistency in the number of selected dimensions.
29
36
We
Contribution 2: A PAC bound on the generalization of SPoSE and VICE models be used retrospectively to provide guarantees about the generalization performance of converged model. Furthermore, it can be used prespectively to determine the sample size required to identify a representation given the number of objects and maximum possible number of dimensions.
This bound can
Contribution 3: Extensive evaluation of model performance across multiple datasets We compare VICE with SPoSE over three different datasets. One of these datasets contains concrete objects, another consists of adjectives, and the third is composed of food items. Experimentally, we find that VICE rivals or outperforms the performance of SPoSE in modeling human behavior. Moreover, we find that VICE yields dimensions that are much more reproducible, with a lower variance for the number of (selected) dimensions. Both of these measures are particularly important in the cognitive sciences. Lastly, we compare VICE with SPoSE for reduced amounts of data and show that VICE has significantly better performance on all measures.
2

---

A PyTorch implementation of VICE featuring Continuous Integration is publicly available at https //github. com/Lukashut/Vrot The GitHub repository additionally includes code to reproduce all of the experiments that are presented in this paper.
2 Related work
Navarro Griffiths introduced a method for learning semantic concept embeddings from item similarity data. The method infers the number of embedding dimensions using the Indian Buffet Process (IBP) 18] Their approach relies on continuous-valiin
similarity ratings rather than
discrete forced-choice behavior and is not directly applicable to our setting. is also challenging
to scale the IBP to the number of dimensions and samples in our work [37]. Roads & Love [39] introduced method for learning object embeddings from behavior in 8-rank-2 task. Their method
predicts behavior from the embeddings by using active sampling [16] to query subjects with the
most informative stimuli, yielding an object similarity matrix. The interpretability of embedding dimensions was not considered this work.
A different approach has been to develop interpretable concept representations from text corpora. Early methods used word embeddings with positivity and sparsity constraints [1 Later work in this direction used topic model representations of Wikipedia articles about objects 335 transformations
of word embeddings into sparse non-negative representations 42 33 or predictions of properties
10 or dimensions [45. Others have considered using text corpora in conjunction with imaging data
13
Finally, Derby et al. []] introduced a neural network function that maps the sparse feature
space of semantic property norm to the dense space of a word embedding, identifying informative combinations of properties and ranking candidate properties for new words.
Triplet task
The triplet task, also known as the triplel odd-one-out task, is used for discovering object con- cept embeddings from similarity judgments over a set of m different objects. These judgments collected from human participants who given queries that consist of triplet of ob- jects (c.g., {"suit", "flamingo", "car")). Participants are asked to consider the three pairs in a and to decide which pair is the most similar, leaving the third as the odd-one-out. We assign each object a numerical index, e.g.. 1 + "aardvark". ..... 1854 + "zucchini". Let (y. 2 denote the indices in this pair, e.g., { = (268, 609) for "suit" and "flamingo." A dataset is set of n ordered pairs of presented triplets and selected pairs. That is, D = (Li,, ka), (Us, (us,22)) )x)) a-1² where {y., 2,} C {is, k.). In Appendix Wewe we discuss in detail why the triplet task appears be a sensible choice for modeling object similarities in humans.
triplet {{"suit", "flamingo"), { {"suit", "car"), " ("flamingo"; "car")), "arrringg""""rr"]
DINAR STANT
Tire
Figure 1: Example triplet containing the objects "suit", "flamingo", and "car" (creative commons images).
4 Formal setting
Sparse Positive object Similarity Embedding (SPoSE) [46] is an approach for finding interpretable embedding dimensions from the triplet task. It does so by finding embedding vector x; = [ra,..., , Fna] for every object i. Let X denote the m d matrix (x1,..., xm) and = X X be the similarity matrix, where S.y denote its entry at j. The probability of choosing {ve, 2g } the most
3

---

similar pair of objects, given an object triplet (se ka) and the embedding matrix X, is modeled as exp(Sy....)
pl 2 (i,.
(1)
exp(St....) + explSt, k..)) + exp(S).,k)
At this point we remark that each triplet (is, Ja: k.) is chosen uniformly at random from the collection Lid. of all possible sets of object triplets T. That is, {úr, J1, 1 k1 ).,,,.., (in:Ja ki ~ U(T). This is precisely stated in Appendix [B.1] SPoSE uses maximum posteriori (MAP) estimation with Laplace prior under non-negativity constraint (equivalent to an exponential prior) to find the most likely embedding X for the training data D. This leads the training objective
TR
arg min (Dg)(()))
X
where log (X constant Σ log ( (3s, 2g H.de ) x) (see Appendix B.1 B.1 derivation the log-likelhhood function) and is determined using cross-validation.
for full
VICE
mean-field VI instead of MAP estimate for approximating the posterior probability p(X (D). In VICE we impose additional constraints on the embedding matrix X by using a prior that encourages shrinkage while allowing entries in X be close to zero.
In contrast SPoSE,
Algorithm 1 VICE optimization for individual triplets for single training epoch
Input: D,0,a = (,,0}
Recall that D = (ei } {yes 2g ) and a is learning rate
Di μr, E Rmxd
for s E nI....,n] do
N(0,1)
Draw i.i.d Gaussian noise where e mm and Esj > Apply reparameterization trick; @ is the Hadamard product Compute similarity matrix using the non-negative parts of Xa.e exp(S
iid. NO 1)
Xo. == +
S = [Xo.d+(Xo.e]
Laaia (Xo,c) & log exp(Si.. texpp.................... (eepp)))xxxppp......e
b Data log- ligeliiklhoood term
Commplexity (Xo,c)
[ [o (Xo.)) - log (ogg((X..) (Xx.c)
KL ddivergennccRRgguulazztiioo term
"
Lwaal (x,c) A Ccomplexity (Xo.c) Ldat (x,.)
t 11 V (X0,c)
Update the embedding means b Update the embedding standard deviations
- a V. Lrotal (Xa,)
end for Output: 0
Return the optimized set of model parameters
5.1
Variational Inference
For VICE we consider approximating (DX with a variational distribution, qo (X), where go E Q and 0 is optimized to minimize the KL divergence the true posterior, p)X In VICE the KL divergence objective function (derived in Appendix [B.2 is
arg min Ey (X)
(log go (X) - log (X)
n Σ log P ((ve, =e)l ka)
(2)
Variational distribution VI requires a choice of parametric variational distribution g E Q. For VICE we use Gaussian distribution with diagonal covariance matrix go (X) =N(H,, diag(o²)) where 0 = [14,a}. o.. Therefore, each embedding dimension has mean and a standard deviation. We deem Gaussian variational distribution appropriate for a variety of reasons. First, under certain conditions, the posterior is Gaussian in the infinite-data limit [24] Second, a unimodal variational distribution makes it possible to use /1,...,, Mm as representative object embeddings. A fixed representation is useful for downstream use cases that rely on a single embedding vector for each object. Use cases range from using embeddings as targets in regression task over unsupervised

---

clustering of embeddings to interpreting embeddings. More complex variational families are clearly not as practical for this. Third, Gaussian posterior computationally convenient choice.
Similarly to Titsias & I Aizai-Grreillla 43 we use a Monte Carlo approximation of Equation 2 by sampling limited number of Xs from (o(X) during training. We generate X with the reparameteri- zation trick [23 38) Xe. μ + @ where E gm is entrywise N 1), and @ denotes the Hadamard (element-wise) product. This leads to the objective
T1
arg min Σ nR
log gX (Xx,c(()))))))
Σ
log Loggl(us (is hi,) (Xo,cte)+))
We apply ReLU function, denoted by [-]+, to the sampled Xg. values to guarantee that Xg.c E
R xd As commonly done in the Dropout and Bayesian Neural Network literature [41 14 27).
set R 1 for computational efficiency during the optimization process. The optimization is outlined for individual triplets (i.e., where B = 1) for a single training epoch in Algorithm
Posterior probability estimation Computational efficiency inference time is not as critical it during training. Therefore, we can get better estimate the posterior probability distribution over the three possible odd one-one-out choices by letting R > 1. Using the optimized variational posterior, ( we approximate the probability distribution with a Monte Carlo estimate from R samples ((r) Xg.ete) for ..... R, yielding
1
p(y. (1..,))) Σ
(4)
R
Spike-and-slab prior As discussed above, SPoSE induces sparsity through an & penalty which, along with the non-negativity constraint, is equivalent using exponential prior. Through examination of the publicly available histograms of weight values in the two most important SPoSE dimensions (see Figure in Appendix (C3)), we observed that the dimensions did not resemble exponential distribution. Instead, they contained spike of probability at zero and wide slab of probability for
the non-zero values. To model this, we use spike-and-slad Gaussian mixture prior (5 12 21 26)
p(X)
TTppikeN((Xijj 0, o spkee + (1 Tppike W O, o
(5)
==1==1
which encourages shrinkage. This prior has three parameters, aspike: dslab; and Typike- Tspike is the probability that an embedding dimension is drawn from the spike Gaussian. Since spike and slab distributions mathematically interchangeable, by convention require that spike < Gslab-
5.2 Dimensionality reduction and convergence
For interpretability purposes it is desirable for the object embedding dimensionality, d. to be small. In contrast to SPoSE, which employs user-defined threshold to prune dimensions, VICE exploits the uncertainty estimates for embedding values to select a subset of informative dimensions.
The pruning procedure works by assigning importance scores to each of the d dimensions, which reflect the number of objects that we can confidently say have a non-zero weight dimension. To compute the score, we use the variational embedding for each object i and dimension location Mi and scale oiy parameters zero according to the left tail of Gaussian distribution with that location and scale (see 1511). This gives us posterior probability of the weight being zero for each object within a dimension [17] To calculate the overall importance of a dimension, we estimate the number of objects that have non-zero weights, while controlling the False Discovery Rate 22 with a 0.05. We define the importance of each dimension to be the number of objects for which P(X.; > 0) > .95 holds. After convergence, we prune the model by removing dimensions with 5 or fewer statistically significant objects. This is a commonly used reliability threshold in semantic property norms (e.g.. (8,110).
compute the posterior probability that the weight is truncated to
In gradient- brade optimization, the gradient of an objective function with respect to the parameters of a model, VL()), is used to iteratively find parameters 0 that minimize that function. We use representational stability criterion to determine convergence. That is, the optimization process
5

---

halts when the number of identified dimensions as described above has not changed by a single dimension over the past L epochs (e.g., 500). Given that our goal is find stable estimates of the number of dimensions, we considered this to be more appropriate than other convergence criteria such as evaluating the cross-entropy error on a validation set or evidence-based criteria [25] For further details on convergence and the optimization process see Appendix C.T
6 Sample complexity bound
We statistical learning theory to obtain estimates of the sample size needed appropriately con- strain VICE (and SPoSE) models. These estimates can be used retrospectively to obtain probabilistic guarantees on the generalization of a trained model unseen data, or prospectively decide how much data to collect for a study. The bound presented here applies to any embedding matrix, X, using the model in Equation [T When using this bound to analyze VICE, we extract ft and use it as a fixed embedding to predict the most likely odd-one-out for query triplet, rather than sampling from the variational distribution.
To obtain a useful bound we make two assumptions. First, we assume that there exists an upper bound M on the largest value in the embedding. Second, we assume that the embeddings obtained by either SPoSE or VICE can be quantized coarsely with marginal losses in predictive performance. We can choose discretization scale A, c.g., A 0.5, and round embedding values to a non-negative integer multiple of A. While we employ this quantization primarily to use learning theory bounds for finite hypothesis classes, it could have benefits for interpretation as well (e.g., dimension could consist of labels such as zero (0), very low (0.5), low (1), medium (1.5), high (2)). In all datasets we have used, SPoSE and VICE embeddings with reasonable priors had values below 2.7 (see Appendix F.2. Given an upper bound M and a discretization scale A, all entries of X are limited to the set A = (0, A, ..... (k- - 1) }A). The following bound tells us that, given enough samples, the true error rate is not much worse than the estimated error rate for such discretized X matrices, with high probability.
Proposition 6.1. Given 8 > O, c > O, and 2 (mdlog(( (k 1) + log (1/8)) / (2c2) training samples ((ys, }.), Iss. kis Y== then
P sup XEAmXa
R(X)- R(X) <() 21 1-55
where
R((): = n
1 Σ 1
(Ver2s) # arg max ((.)) (v.z)
and
=(X)== = P( (V/ & arg argmaxp (Lv, ((0,))(00, /
(v.=)
Proof sketch. This bound follows from applying Hoeffding's s inequality to the (A- + 1)mxd elements of A mxd combined with a union bound. This proof is virtually identical to that of Theorem 2. 13 in Mohri al. 300 A complete proof can be found in Appendix D
To use the bound we first decide on the number of quantization steps k = [M/A]. The probability of violating the bound, ö, is analogous the Type error control in hypothesis testing and is often set to 0.05. Since n in our proposition depends very weakly on ó, it is convenient to use 8 = 0.001. The error tolerance does have major effect on the sample size estimate provided by the bound. For the bound to be practically useful, e has to be smaller than the difference between the training error average zero-one loss over the training set examples and random guessing error. For the datasets we use in this paper we observed a difference of 0.2 - 0.3. A conservative choice would (from above), these values result in prospective sample size of n 2 50 (og + log( 1000)). To use the bound retrospectively. we fix n. 8 while varying A (and consequently k) to get guarantee on c, which in turn yields probabilistic upper bound on the generalization error for an embedding, R(X)= S R((X) + e. For more details, see Algorithm in Appendix F.1
halve the accuracy gap of 0.2, giving c 0.1. Together, with

---

Divergence
Dimens lonality
Accura
% data usdd training
15.00 data used for traniing
35 data used training
Figure 2: VICE vs. SPoSE. Left: KL divergences between model output and human probability distributions. Center: Triplet task prediction accuracies. Right: Number of identified embedding dimensions. VICE and SPoSE were each trained on differently sized subsets the THINGS training data. Error bands visualize 95% across the different random initializations and partitions of the data.
7
Experiments
Data We performed experiments for three triplet datasets: THINGS (used to develop SPoSE 46 19 ADJECTIVES and FOOD [7] THINGS and ADJECTIVES contain random samples from all possible triplet combinations for 1,854 objects and 2,372 adjectives, respectively. THINGS and ADJECTIVES each comprised of large training dataset which contains no "repeats," i.e., there is only one human response for each triplet (~ 1.5M triplets for THINGS and ~ 800K triplets for ADJECTIVES). predefined validation set. The
For each of these training datasets, 10% of the triplets are assigned test sets for both contain the results for 1,000 random triplets that are not contained in the training data, with 25 repeats for each triplet. FOOD contains data for every possible triplet combination for 36 objects and repeats for some triplets. We partitioned this dataset into train (45%), validation (5%), and test (50%) sets with disjoint triplets. Appendix G provides details about stimuli, data collection, and quality control.
Experimental setup We implemented both SPoSE and VICE in PyTorch 34 using Adam (2 with 17 = 0.001. To find the best VICE hyperparameter combination, we performed a grid search spike: saaa, spiike: optimizing Equation 3 and evaluating each model on the validation set. All experiments were performed over 20 different random initializations. For VICE we observed that the hyperparameter values with the lowest average cross-entropy error the validation similar across the different datasets (see Table i Appendix E For more information about the experimental setup, including details about training, weight initialization, hyperparameter grid, and optimal combination, see Appendix D
highly
Evaluation measures VICE estimates p(ly. ()), the model's softmax probability dis- tribution over a given triplet (see Equation 4. The first evaluation measure we consider is the prediction accuracy with respect to the correct human choice <y.2), where the predicted pair is arg max (p.z) å (v, H, k)). We estimate an upper bound on the prediction accuracy by using the repeats in the test set (see Appendix E}. A Bayes-optimal classifier would predict the majority outcome for any triplet, whereas random guessing gives accuracy of 1/3. Note that the triplet task subjective and thus there is not necessarily a definitive correct answer for a given triplet. If a test set contains multiple responses for triplet, it provides information about the relative similarities of the three object pairs. This lets us calculate probability distributions over answers for each triplet. Approximating these distributions closely is important in cognitive science applications. Therefore, we additionally evaluate the models by computing the KI divergences between the models' posterior probability estimates ý (v. =) {i, j, k)) (see Equation 4 and the human probability distributions inferred from the test set across all test triplets and report the mean.
7.1 Results on THINGS
Full dataset For this experiment compared representative model from VICE and SPoSE. The representative models were chosen to be those with the median cross-entropy error on the validation set over the 20 seeds. For VICE, we set the number of Monte Carlo samples R 50. On the test set, VICE and SPoSE achieved similar prediction accuracies of 0.638 and 0.637 respectively (estimated upper bound for accuracy was 0.673). Likewise, VICE and SPoSE achieved similar
AdJeectVvss is not yet puhlished, but was shared with us by Shruti Japce.
FOOD was shared with by Jason Avery.
7

---

average KL divergences of 0.100 and 0.103 respectively (random guessing KL divergence was 0.366). The differences between the median model accuracies and KL divergences were not statistically significant according to two-sided paired /-test over individual test triplets. Hence, VICE and SPoSE predicted triplets approximately equally well when trained on the full dataset. This is not surprising since Bayesian methods based on Monte Carlo sampling become more like deterministic maximum likelihood estimation in the infinite data limit. If n + oo, then the left-hand side in the expectation of Equation 2 which accounts for the contribution of the prior, goes to zero. Conversely, the effects of the prior are more prominent when n is small, as we show in the following section.
Data efficiency experiments Performance on small datasets is important in cognitive science, since behavioral experiments often have small sample sizes (e.g., tens to hundreds of volunteer in-lab subjects) compared to THINGS, which is particularly large, or they can be costly to collect. To test whether VICE can model the data better than SPoSE in small sample regimes, we performed experiments training on smaller subsets with sizes equal to 5%, 10%, 20%, and 50% of the training dataset. For each size, we divided the training set into equal partitions and performed an experiment for each partition, thereby giving multiple results for each subset size. Validation and held-out test sets remained unchanged. In Figure 2 we show the average KL divergence (left) and prediction accuracy (middle) across data partitions and 20 random seeds (used to initialize model parameters) for various dataset sizes. The average performance across partitions was used to compute the confidence intervals (CIs). The differences in prediction accuracies and KL divergences between VICE and SPoSE were notable for the 5% and 10% data subsets. In the former, SPoSE predicted only slightly better than random guessing; in the latter, SPoSE showed large variability between random seeds and data partitions, as can be seen in the 95% Cls. In comparison, VICE showed small variation in the two performance metrics across random seeds and performed much better than random guessing. The differences between VICE and SPoSE for the 5% and 10% subset scenarios were statistically significant according to a two-sided paired f-fest comparing individual triplet predictions between median models (p < 0.001). For the full training dataset, VICE used significantly fewer dimensions (p 0.001; unpaired sign test) than SPoSE achieve comparable performance.
7.2 Other results
Other datasets On the ADJECTIVES test set, VICE and SPoSE had accuracies of 0,559 and 0.562, respectively (estimated upper bound was 0.607), and KL divergences of 0.083 and 0.088, respectively (random guessing was 0.366). On the FOOD test set, VICE and SPoSE median models had accuracies of 0.693 and 0.698, respectively. The differences in accuracy (and KL divergence for ADJECTIVES) between VICE and SPoSE for both datasets were not statistically significant according to two-sided paired f-tests on the median model predictions across triplets on the test set. However, for FOOD, VICE used significantly fewer dimensions than SPoSE to achieve similar performance (see Table [
Hyperparameters We observed that VICE's performance fairly insensitive to hyperparameter selection. Using default hyperparameters, Sspike = 0.25, stab = 1.0, Tsigma 0.5, yields an accuracy score within 0.015 of the best cross-validated model in the full dataset setting, on all three datasets.
7.3 Reproducibility and Representational stability
Reproducibility As mentioned in the introduction, key criterion for learning concept representations beyond predictive performance is reproducibility, i.e., learning similar representations for different random initializations on the same training data. To evaluate this, we compared the learned embed- dings from 20 differently initialized VICE and SPoSE models. The first aspect of reproducibility we investigate is whether the models yield consistent number of embedding dimensions across random seeds.
As reported in Table VICE yielded fewer dimensions than SPoSE with less variance in the number of dimensions for all three datasets. The embedding dimensionality was significantly smaller according to an independent t-test for THINGS (p < 0.001) and FOOD (p < 0.001), but not for ADJECTIVSS (p 0.108). The difference in the standard deviations for ADJECTIVES was statistically significant according to a two-sided F-test (p- 0.002), but was not statistically significant for THINGS (p- = 0.283) or FOOD (p- = 0.378). In addition, we observed that the identified dimensionality is consistent long d is chosen to be sufficiently large (see Appendix E.
8

---

Table 1: Reproducibility of VICE and SPoSE. Reported the and standard deviations with respect
selected dimensions and average reproducibility score of dimensions (in %) across random seeds. Bold
means VICE performed statistically significantly better with a = 0.05.
VICE
SPOSE
DATA METRIC
Selected Dims.
Reproducibility
Selected Dims,
Reproducibility
THINGS
44 (1.59)
87.01%
52 (1.82)
81.30%
ADJECTIVES
21 (0.77)
76.76%
22 (1.53)
71.64%
FOOD
(0.95)
87.38%
16 (1.02)
62.88%
The second aspect to reproducibility we examine the degree to which the identified dimensions are similar across random initializations to a permutation. To calculate the number of reproducible dimensions, we associated each embedding dimension of model with the most similar embedding dimension across the other models. We quantify reproducibility of dimension the average Pearson correlation between one dimension and its best match across the remaining models. In Table [ report the average relative number of dimensions with correlation 0.8 across models. The embedding dimensions in VICE were more reproducible than those SPoSE. The difference in average reproducibility scores was statistically significant according to two-sided, independent t-test for FOOD (p- = 0.030) and THINGS (p 0.008), but not for ADJECTIVES (p = 0.466).
Representational stability For all three datasets, VICE found reproducible and stable dimensions across all 20 random initializations and converged when the embedding dimensionality had not changed over the past == 500 epochs. The median number of epochs to achieve representational stability was comparable for the similarly-sized THINGS and ADJECTIVES datasets, but occurred later for FOOD (likely due the smaller number of gradient updates per epoch). In Appendix C.I we show plots that demonstrate that the convergence criterion as defined in $5.2 worked reliably for all datasets.
7.4 Interpretability
THINGS
Food
20 Bigantt Tc giant
str striped tooo THE lagged HOLLEE
norridbad despicable Groublesces hurtful
moistjuica mushy CP
Adjectives
oversized Sy Ligelarge maxinum
milky 01
soapy
thin ymme trical
offfens www.eieeeaaa aaru
Figure 3: Four example VICE dimensions showing the top six objects for THINGS (top) and FOOD (middle), and wordclouds for ADJECTIVES (bottom).
One of the benefits of SPoSE is the interpretability of the dimensions of its concept embeddings which was evidenced through experiments with humans [19]. VICE dimensions are equally interpretable due to similar constraints on its embedding space. Therefore, it is easy to sort objects within dimension of the VICE mean embedding matrix f descending order, obtain human judgments of what an embedding dimension represents. To illustrate the interpretability of VICE we show in Figure four example dimensions of representative model for THINGS. FOOD, and ADJECTIVES. For THINGS the dimensions appear represent calegorical, functional, struciural, and color-related information. For FOOD the dimensions appear to be a combination of fried, vegetable, fruit, and sweet food items. For ADJECTIVES the dimensions seem to reflect size/magnitude, visual appearance.

---

negative valence, and sensory adjectives. In Appendix H we show VICE dimensions for THINGS only, since manuscripts about the other datasets are still preparation, and, hence, cannot be shown publicly.
8 Limitations
The goal of our method is to identify general mental representations of objects. The dimensions identified by a model reflect semantic characteristics that explain task performance for many subjects in the experimental subject population (here, Amazon Mechanical Turk subjects in the United States). As such, it is possible that they reflect biases widespread in that population. Furthermore, the choice of population may affect the identified dimensions. That is, a chéf may classify food items differently from lay subject, and a linguist would likely have a more complex representation of an adjective. The effects of expertise or developmental stage in mental representations are of obvious interest to cognitive scientists. Therefore, we envision further research in those areas, which may additionally provide some indication of how widely representations can vary.
9
Conclusion
One of the central goals in the cognitive sciences is the development of computational models of mental representations of object concepts. Such models may serve as component for other behavioral prediction models or as basis for identifying the nature of concept representations in the human brain. In this paper we introduced VICE, novel VI approach for learning interpretable object concept embeddings by modeling human behavior in an triplet odd-one-out task. We showed that VICE predicts human behavior close to the estimated best attainable performance across three datasets and that VICE outperforms competing method, SPoSE, in low data regimes. In addition, VICE has several characteristics that are desirable for scientific use. has an automated procedure for determining the number of dimensions sufficient to explain the data, which further enables the detection of convergence during training. This leads to better model reproducibility across different random initializations and hyperparameter settings. As result, VICE be used out-of-the- without requiring to perform an extensive search over random seeds or tuning model hyperparameters. Finally, we introduced a PAC learning bound on the generalization performance for a VICE model. Although VICE assumes a shared mental representation across participants akin to much of the literature ,, believe that the VI framework can be leveraged model inter-individull differences, which we plan to do in future work.
Acknowledgments
LM and RV acknowledge support by the Federal Ministry of Education and Research (BMBF) for the Berlin Institute for the Foundations of Learning and Data (BIFOLD) (011S18037A). LM and MNH acknowledge support by a Max Planck Research Group grant awarded to MNH by the Max Planck Society. FP, CZ, and PM acknowledge the support of the National Institute of Mental Health Intramural Research Program (ZIC-MH002968), PM acknowledges the support the Naval Postgraduate School's Research Initiation Program. This study utilized the high-performance computational capabilities of the Biowulf Linux cluster the National Institutes of Health, Bethesda, MD (http://biowull..nb) gov) and the Raven and Cobra Linux clusters at the Max Planck Computing & Data Facility, Garching, Germany (https://www. wwt.wppp...mmm.m mpg. www../ppde/eeeiee//ssueeeeeeeeee www.seeviiee/suprrnoooooo te/ssevvcessuuprccopuutnnn)))))) The authors would like to thank Chris Baker for useful initial discussions, Shruti Japee, Jason Avery, and Alex Martin for sharing data, and Erik Daxberger, Lorenz Linhardt, Adrian Hill, Niklas Schmitz, and Marco Morik for valuable feedback earlier versions of the paper.
References
[1] Alexandre Belloni and Victor Chernozhukov. Least squares after model selection in high- dimensional sparse models. Bernoulli, 19(2)5211-547 2013. doi: 10.3150/11-BEJ410.
[2] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Method- ological), 57(1):289 1995. doi:
10

---

[3] Jeffrey R Binder, Lisa L Conant, Colin J Humphries, Leonardo Fernandino, Stephen B Simons, Mario Aguilar, and Rutvik H Desai. Toward a brain-basee componential semantic representation. Cognitive Neuropsychology.
((33-4):330-17) 2016.
[4] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):59-887) 2017. doi: (00000006214992217...
[5] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Leurning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1613-1622. JMLR.org. 2015.
[6] Erin M. Buchanan, K. D. Valentine, and Nicholas P. Maxwell. English semantic feature production norms: An extended database of 4436 concepts. Behavior Research Methods, 51(): 1849-1863, May 2019. doi:
[7] Madeline Carrington, Alexander G. Liu, Charles Y. Zheng, Lukas Muttenthaler, Francisco Pereira, Jason Avery, and Alex Martin. Naturalistic food categories are driven by subjective estimates rather than objective measures of food qualities. In Society for Neuroscience Annual Meeting, Washington DC., 2021.
[8] Steven Derby, Paul Miller, Brian Murphy, and Barry Devereux. Using sparse semantic embed- dings learned from multimodal text and image data to model human conceptual knowledge. In Anna Korhonen and Ivan Titov (eds.), Proceedings of the 22nd Conference on Computational Natural Language Learning, CoNLL 2018, Brussels, Belgium, October 31 November 1, 2018, Association for Computational Linguistics, 2018. doi: 10. 10.18653/w1/k18 1026.
pp. 260-270.
[9] Steven Derby, Paul Miller, and Barry Devereux. Feature2vec: Distributional semantic modelling of human property knowledge. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 5852-5858. Association for Computational Linguistics, 2019, doi: 1/D16
[10] Barry J. Devereux, Lorraine K. Tyler, Jeroen Geertzen, and Billi Randall. The centre for speech, language and the brain (CSLB) concept property norms. Behavior Research Methods, 46(4): 1119-1127, December 2013, doi: 10.37558/13422--03-002222
[11] David Duvenaud, Dougal Maclaurin, and Ryan Adams. Early stopping as nonparametric variational inference. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. 1070-1077, Cadiz, Spain, 09-11 May 2016. PMLR.
[12] Ludwig Fahrmeir, Thomas Kneib, and Susanne Konrath. Bayesian regularisation in structured additive regression: A unifying perspective shrinkage, smoothing and predictor selection. Statistics and Computing, 20(2):203-211 2199, 2010. doi: 10.10/7/s 009-9599-..
[13] Alona Fyshe, Partha Pratim Talukdar, Brian Murphy, and Tom M. Mitchell. Interpretable semantic vectors from a joint model of brain- and text-based meaning. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 1: Long Papers, pp. 489-499. The Association for Computer Linguistics, 2014. doi: 10.3115/v1/p141000
[14] Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In Mloriaal Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1050-1059. JMLR.org. org,, 2016.
[15] Edward I. George and Robert E. McCulloch. Variable Selection via Gibbs Sampling. Journal of the American Statistical Association, 88(423):881-$$5 1993, doi: 10. 1000000001224400 (10.22111919999 1993. 10476353.

---

[16] Jacqueline Gottlieb and Pierre-Yve Oudeyer. Towards a neuroscience of active sampling and curiosity. Nature Reviews Neuroscience, 19(12):758-77) November 2018. doi: 10.1038/
00883-00--008-0..
[17] Alex Graves. Practical variational inference for neural networks. In John Thylor, Richard S. 7emel, Peter I.. Bartlett, Fernando C. N. Pereira, and Kilian Q Weinberger (eds.), Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of meeting held 12-14 December 2011, Granada, Spain, pp. 2356,, 2011.
[18] Thomas L. Griffiths and Zoubin Ghahramani. The Indian Buffet Process: An Introduction and Review. Journal of Machine Learning Research, 12:1 185-1224, 2011.
[19] Martin N. Hebart, Charles Y. Zheng, Francisco Pereira, and Chris I. Baker. Revealing the multi- dimensional mental representations of natural objects underlying human similarity judgements. Nature Human Behaviour, 4(11):1 October 2020. doi: 10. 10.10388<41552--000000999
[20] Mariam Hovhannisyan, Alex Clarke, Benjamin R. Geib, Rosalic Cicchinelli, Zachary Monge, Tory Worth, Amanda Szymanski, Roberto Cabeza, and Simon W. Davis. The visual and semantic features that predict object memory: Concept property norms for 1, 000 object images. Memory & Cognition, (20()))212-31)) January 2021. doi: 11227780/11421--2000000
[21] Hemant Ishwaran and J. Sunil Rao. Spike and slab variable selection: frequentist and bayesian strategies. The Annals of Statistics, 33(2):730-773, 2005.
[22] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, Sun Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[23] Diederik P. Kingma and Max Welling. Anto-Encoding Variational Bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[24] Bas JK Kleijn and Aad W van der Vaart. The Bernstein-von-Misss theorem under misspecifica- tion. Electronic Journal of Statistics, 6:354-381, 2012.
[25] Maren Mahsereci, Lukas Balles, Christoph Lassner, and Philipp Hennig. Early stopping without a validation set. arXiv e-prints, art. arXiv:1703.09580, March 2017.
[26] Gertraud Malsiner- and Helga Wagner. Comparing spike and slab priors for bayesian variable selection. arXiv e-prints, art. arXiv:1812.07229, December 2018.
[27] Patrick McClure and Nikolaus Kriegeskorte. Robustly representing uncertainty in deep neural networks through sampling. arXiv e-prints, art. 1.Xiv:1611.0166 November 2016.
[28] Ken McRae, George S. Cree, Mark S. Seidenberg, and Chris Mcnorgan. Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37 (4):547-559, November 2005. doi: (..758/b603192722
[29] J. Mitchell and J. Beauchamp. Bayesian variable selection in linear regression. Journal of the American Statistical Association, 83(404) 1032, 1988. doi: 10. 10.1080001621559.1998 10478694.
[30] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. Adaptive Computation and Machine Learning Series. The MIT Press, 2018
[31] Brian Murphy, Partha Talukdar, and Tom Mitchell. Learning effective and interpretable semantic models using non-negative sparse embedding. Proceedings of COLING 2012, pp. 1933-1950, Mumbai, India, December 2012. The COLING 2012 Organizing Committee.
[32] Daniel Navarro and Thomas I. Griffiths. Latent features in similarity judgments: nonpara- metric Bayesian approach. Neural Computation, 20(11))2597-2688 2008
12

---

[33] Abhishek Panigrahi, Harsha Vardhan Simhadri, and Chiranjib Bhattacharyya. Word2sense: Sparse interpretable word embeddings. In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 5692-5705. Association for Computational Linguistics, 2019. doi: 10. 19956/191/111.
[34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Bus Buch Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8024-8035, 2019.
[35] Francisco Pereira, Matthew Botvinick, and Greg Detre. Using Wikipedia to learn semantic feature representations of concrete concepts in neuroimaging experiments. Artificial Intelli- gence, 194:240-252, 2013. ISSN 0004-3702. doi: https://doi.org/100 101. Artificial Intelligence, Wikipedia and Structtrut Resources.
[36] Kolyan Ray, Botond Szabó, and Gabriel Clara. Spike and slab variational Bayes for high dimensional logistic regression. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[37] Colorado Reed and Zoubin Ghahramani. Scaling the Indian Buffet Process via Submodular Maximization. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pp. 1013-1021. JMLR.org, 2013.
[38] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Eric P. Xing and Tony Jebara (eds.), Pro- ceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1278- 1726,, Bejing, China, 22-24 Jun 2014. PMLR.
[39] Brett D Roads and Bradley Love. Enriching ImageNet with human similarity judgments and psychological embeddings. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3547-3557, 2021.
[40] Veronika Rocková and Nicholas Polson. Posterior concentration for sparse deep learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 938-949, 2018
[41] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-195) 1958, 2014.
[42] Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Barg-Kirkpatrick, and Eduard H. Hovy. SPINE: sparse interpretable neural embeddings. In Sheila A. Mcllraith and Kilian Q- Weinberger (eds.), Proceedings of the Thirly-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 pp. 4921-4928. AAAI Press, 2018
[43] Michalis Titsias and Miguel Eizaro-Gredilla Doubly stochastic variational bayes for non- conjugate inference. In Eric P. Xing and Tony Jebara (eds.). Proceedings of the 31 International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1971-1979, Bejing, China, 22-24 Jun 2014. PMLR
13

---

[44] Michalis K. Titsias and Miguel Lázaro-Green Gredilla. Spike and Slab Variational Inference for Multi- Task and Multiple Kernel Learning. John Shawe- Taylor, Richard S. Zemel, Peter L. Bartlett, Fernando C. N. Pereira, and Kilian Q Weinberger (eds.), Advances in Neural Information Processing Systems 24: Annual Conference on Neural Information Processing Systems 201 I, NeurlPS 1011, December 12-14, 14, 2011, Granada, Spain, pp. 2339-2347, 2011.
[45] Akira Utsumi. Exploring what is encoded in distributional word vectors: A neurobiologically motivated analysis. Cognitive Science, 44(6), 2020. doi: 10.1111/cogs.12880 12844.
[46] Charles Y. Zheng, Francisco Pereira, Chris I. Baker, and Martin N. Hebart. Revealing in- terpretable object representations from human behavior. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)? [Yes] See https //github. wttplu/...ooouuuaa/ttt///////
for our codebase. See https://osf /o/ jum2f/ and httpp::/thingg-iiittttee instiatiiiiiiiie ortttiieee.rr//oooooeeee for data.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See $
7 7 and Appendix B
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See Figure and Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
See https: ptttps://githul
(d) Did you discuss whether and how consent obtained from people whose data you're
using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Appendix G
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
We re-used crowdsourced data collected by various collaborators, and attribute it to them in the main paper. Even though we did not collect these data ourselves, we provide summary of the information asked for in 5. in Appendix G as well as a short description of the task and stimuli in Section
15


Reviews:
[{"review_id": "VjRUZq88Vtp", "reviewer": "Reviewer_fyj5", "summary": "This paper introduces VICE, a variational inference method for embedding object concepts into a vector space so as to obtain sparse and non-negative representations of them. VICE follows the same approach as its predecessor (SPoSE) to model the odd-one-out triplet task, but addresses SPoSE's major limitations by taking advantage of recent VI techniques. In addition, it also derives a PAC learning bound for SPoSE and VICE models, which can be used to estimate generalization performance or determine sufficient sample size.", "questions": "(1) The main difference from SPoSE need to be highlighted, technically.\n", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\nThe paper is overall clearly written and well-organized. The experiment section clearly states the setup/evaluation protocol and presents detailed analyses.\n\nCons:\n\nOverall, I think the novelty of this paper is limited. Essentially, the idea of learning a sparse, positive (and hence interpretable) semantic space to be consistent with human similarity judgements (or finish the odd-one-out triplet task) is no different from SPoSE (by Charles et al). Following the same modeling process, VICE just makes some improvements over SPoSE with the help of VI techniques. The choice of Gaussian variational distributions for ease of reparameterization and the use of a spike-and-slab prior to induce sparsity are also based on previous successful experiences, so the ideas are not new there. It seems that the real novelty comes more from the derivation of a PAC bound on the generalization of SPoSE and VICE models, but I’m not sure I understand it since the author does not present the proof clearly enough.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G_kN0Pqkzeb", "reviewer": "Reviewer_VDhE", "summary": "The paper proposed VICE, a way to learn interpretable embeddings that mimic that of a human. The paper is different from its main baseline (SPoSE) by using variational inference with a \"spike and slab\" Gaussian prior. The paper improves upon SPoSE by not having a hyperparameter that finds the # of embedding dimensions to ensure sparsity. VICE is able to improve upon SPoSE in the low-data regime, which the paper argues, is the most relevant setting in cognitive science.", "questions": "Please see above.", "limitations": "The authors do not discuss any societal limitations. It would be good to discuss if in the compression process, if it is possible that the model is implicitly encoding any biases that may exist in the data, since its directly labeled by humans, and there is not an absolute \"ground truth\" available for such datasets.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The paper seems theoretically sound as it improves considerably over the baselines\n- The paper is written and explained well and has a clear story\n- The results in the low data regime are very convincing as the model improves performance by ~20% in the lowest-data regime.\\\n- The dimensionality reduction procedure and spike-and-slam priors used by the paper is interesting.\n\nWeaknesses:\n- It would be preferred if the paper expanded on Fig 1, to give readers a better intuitive example of what the structure of the dataset is and some gt labels in the dataset, since it isn't obvious which one of the thruple is the \"odd-one-out\"\n- Qualitative results for both SPoSE and VICE would be helpful (instead of just for VICE\n- I am not sure if there are other baselines possible for such a paper (such as sysnet and NNSE used in Zhang et al.)\n- I am not sure what the authors mean by: \"We estimate an upper bound on the prediction accuracy by using the repeats in the test set.\" How is the upper bound in accuracy found?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AcrSijBMF5E", "reviewer": "Reviewer_tLve", "summary": "This work comprehensively analyzed the limitations of SPoSE from the aspects of the posterior distribution, estimation method, and convergence criterion. The proposed method VICE addressed the above limitations with elegant mathematical support. The authors reinterpreted the objective of SPoSE from the perspective of maximizing a posteriori (MAP) estimation, which is insightful for future work. The proofs are solid, especially dimensionality reduction and convergence. The experimental results were also positive to support the claims.", "questions": "- Since it is a supervised method, how about the efficiency of labeling a pair from a triplet? Why not label the concepts directly?\n- Does the Spike-and-Slab prior for p(X) come from an empirical observation?\n- Why the embeddings should be positive?", "limitations": "This method needs numerous labels from human judgments. However, some work could promote dimensional explanation without supervision, such as disentanglement learning. The main limitation is the problem settings in the odd-one-out triplet task. Despite the limitations, I am looking forward to seeing the benefit of supervision and possible approaches to building large triplet datasets by automatic techniques.\n\nReference:\n1. DEFT: Distilling Entangled Factors by Preventing Information Diffusion", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths ：\n+ This work explicitly described the introduced assumptions and how to induce the proposed method. This part is easy to read and understand.\n+ This work originally proposed VICE with a unimodal posterior for representing each object and variational inference for optimization to learn from human judgments about object similarity.\n+ This work encouraged interpretable representations, which is highly desirable for robust and trustworthy AI systems.\n\nWeaknesses：\n- It's unclear for the neural network settings and how to get the embedding vectors.\n- What are the effects of the biases from judgments? It's important but missed. For instance, will the learned representations from ordinary people be different from those from specialists?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "3XHnpwMkIgv", "reviewer": "Reviewer_NZMy", "summary": "In this paper, a model named VICE is introduced for learning interpretable object concept embeddings by modelling human behaviour in an odd-one-out triplet task. It is shown that VICE predicts human behaviour close to the estimated best attainable performance across three datasets and that VICE outperforms a competing method SPoSE, in low sample regimes. In addition, the authors claim that VICE has an automated procedure for determining the number of dimensions needed to explain the data.", "questions": "Please refer to the previous section for detailed comments and discussion.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper is relatively well written and the proposed method is easy to follow and understand.\n2. The proposed method VICE is technically correct, including modelling definition, equations and derivations.\n3. Both qualitative and quantitative comparisons are given to showcase the modelling performance and certain properties of the proposed method.\n4. There is a lot of discussion on the intuition of the design of VICE, and discussion on comparison between VICE and SPoSe. I really enjoyed reading these discussions.\nWeakness:\n1. The paper claims to work on a very novel task of modelling human behaviour in an odd-one-out triplet task. However, after reading the description, it seems like a common classification task. You are given three input, and a set of two as label. It would be informative if the authors can expand the discussion on the task description part.\n2. Related to point 1, it is also not clear how this task is associated with the concept embedding goal being evaluated. It is completely ok if learning a interpretable concept embedding is a nice side benefit of the task, or vice versa, but it would be clearer if the authors discuss this.\n3. The paper needs more proof reading. For example, line 22 \"In an alternative, detective, approach, ...\" it look me a while to figure out the grammar here and to understand this sentence.\n4. Triplet task seems to be a classification task. In line 104, the authors mentioned that the dataset is a set of ordered pairs. Why does the pairs need to be ordered? Does the ordering encode certain information?\n5. Prior choice. Spike-and-Slab is chosen in this work based on analysis and intuition. Since this is a major novelty component of the proposed model, it would be more convincing if the authors could provide some experiments on the choice of prior. Such a comparison would back up the analysis on previous work and support the choice of Spike-and-Slab as prior.\n6. The authors claim that one of the property of VICE is that VICE has an automated procedure for determining the number of dimensions needed to explain the data. Therefore, there isn't much need for running lots of configurations to tune hyper parameter. However, reading section 3.3.2, there is still a large number of hyper parameters. Even if there is a commonly used threshold present, it doesn't erase the fact that these hyper parameters still needs to be tuned to achieve the best modelling performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept