Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposed VICE, a way to learn interpretable embeddings that mimic that of a human. The paper is different from its main baseline (SPoSE) by using variational inference with a "spike and slab" Gaussian prior. The paper improves upon SPoSE by not having a hyperparameter that finds the # of embedding dimensions to ensure sparsity. VICE is able to improve upon SPoSE in the low-data regime, which the paper argues, is the most relevant setting in cognitive science.
strengths_and_weaknesses: Strengths: 
- The paper seems theoretically sound as it improves considerably over the baselines
- The paper is written and explained well and has a clear story
- The results in the low data regime are very convincing as the model improves performance by ~20% in the lowest-data regime.\
- The dimensionality reduction procedure and spike-and-slam priors used by the paper is interesting.

Weaknesses:
- It would be preferred if the paper expanded on Fig 1, to give readers a better intuitive example of what the structure of the dataset is and some gt labels in the dataset, since it isn't obvious which one of the thruple is the "odd-one-out"
- Qualitative results for both SPoSE and VICE would be helpful (instead of just for VICE
- I am not sure if there are other baselines possible for such a paper (such as sysnet and NNSE used in Zhang et al.)
- I am not sure what the authors mean by: "We estimate an upper bound on the prediction accuracy by using the repeats in the test set." How is the upper bound in accuracy found?
questions: Please see above.
limitations: The authors do not discuss any societal limitations. It would be good to discuss if in the compression process, if it is possible that the model is implicitly encoding any biases that may exist in the data, since its directly labeled by humans, and there is not an absolute "ground truth" available for such datasets.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, a model named VICE is introduced for learning interpretable object concept embeddings by modelling human behaviour in an odd-one-out triplet task. It is shown that VICE predicts human behaviour close to the estimated best attainable performance across three datasets and that VICE outperforms a competing method SPoSE, in low sample regimes. In addition, the authors claim that VICE has an automated procedure for determining the number of dimensions needed to explain the data.
strengths_and_weaknesses: Strength:
1. The paper is relatively well written and the proposed method is easy to follow and understand.
2. The proposed method VICE is technically correct, including modelling definition, equations and derivations.
3. Both qualitative and quantitative comparisons are given to showcase the modelling performance and certain properties of the proposed method.
4. There is a lot of discussion on the intuition of the design of VICE, and discussion on comparison between VICE and SPoSe. I really enjoyed reading these discussions.
Weakness:
1. The paper claims to work on a very novel task of modelling human behaviour in an odd-one-out triplet task. However, after reading the description, it seems like a common classification task. You are given three input, and a set of two as label. It would be informative if the authors can expand the discussion on the task description part.
2. Related to point 1, it is also not clear how this task is associated with the concept embedding goal being evaluated. It is completely ok if learning a interpretable concept embedding is a nice side benefit of the task, or vice versa, but it would be clearer if the authors discuss this.
3. The paper needs more proof reading. For example, line 22 "In an alternative, detective, approach, ..." it look me a while to figure out the grammar here and to understand this sentence.
4. Triplet task seems to be a classification task. In line 104, the authors mentioned that the dataset is a set of ordered pairs. Why does the pairs need to be ordered? Does the ordering encode certain information?
5. Prior choice. Spike-and-Slab is chosen in this work based on analysis and intuition. Since this is a major novelty component of the proposed model, it would be more convincing if the authors could provide some experiments on the choice of prior. Such a comparison would back up the analysis on previous work and support the choice of Spike-and-Slab as prior.
6. The authors claim that one of the property of VICE is that VICE has an automated procedure for determining the number of dimensions needed to explain the data. Therefore, there isn't much need for running lots of configurations to tune hyper parameter. However, reading section 3.3.2, there is still a large number of hyper parameters. Even if there is a commonly used threshold present, it doesn't erase the fact that these hyper parameters still needs to be tuned to achieve the best modelling performance.
questions: Please refer to the previous section for detailed comments and discussion.
limitations: yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces VICE, a variational inference method for embedding object concepts into a vector space so as to obtain sparse and non-negative representations of them. VICE follows the same approach as its predecessor (SPoSE) to model the odd-one-out triplet task, but addresses SPoSE's major limitations by taking advantage of recent VI techniques. In addition, it also derives a PAC learning bound for SPoSE and VICE models, which can be used to estimate generalization performance or determine sufficient sample size.
strengths_and_weaknesses: Pros:

The paper is overall clearly written and well-organized. The experiment section clearly states the setup/evaluation protocol and presents detailed analyses.

Cons:

Overall, I think the novelty of this paper is limited. Essentially, the idea of learning a sparse, positive (and hence interpretable) semantic space to be consistent with human similarity judgements (or finish the odd-one-out triplet task) is no different from SPoSE (by Charles et al). Following the same modeling process, VICE just makes some improvements over SPoSE with the help of VI techniques. The choice of Gaussian variational distributions for ease of reparameterization and the use of a spike-and-slab prior to induce sparsity are also based on previous successful experiences, so the ideas are not new there. It seems that the real novelty comes more from the derivation of a PAC bound on the generalization of SPoSE and VICE models, but I’m not sure I understand it since the author does not present the proof clearly enough.

questions: (1) The main difference from SPoSE need to be highlighted, technically.

limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work comprehensively analyzed the limitations of SPoSE from the aspects of the posterior distribution, estimation method, and convergence criterion. The proposed method VICE addressed the above limitations with elegant mathematical support. The authors reinterpreted the objective of SPoSE from the perspective of maximizing a posteriori (MAP) estimation, which is insightful for future work. The proofs are solid, especially dimensionality reduction and convergence. The experimental results were also positive to support the claims.
strengths_and_weaknesses: Strengths ：
+ This work explicitly described the introduced assumptions and how to induce the proposed method. This part is easy to read and understand.
+ This work originally proposed VICE with a unimodal posterior for representing each object and variational inference for optimization to learn from human judgments about object similarity.
+ This work encouraged interpretable representations, which is highly desirable for robust and trustworthy AI systems.

Weaknesses：
- It's unclear for the neural network settings and how to get the embedding vectors.
- What are the effects of the biases from judgments? It's important but missed. For instance, will the learned representations from ordinary people be different from those from specialists?


questions: - Since it is a supervised method, how about the efficiency of labeling a pair from a triplet? Why not label the concepts directly?
- Does the Spike-and-Slab prior for p(X) come from an empirical observation?
- Why the embeddings should be positive?
limitations: This method needs numerous labels from human judgments. However, some work could promote dimensional explanation without supervision, such as disentanglement learning. The main limitation is the problem settings in the odd-one-out triplet task. Despite the limitations, I am looking forward to seeing the benefit of supervision and possible approaches to building large triplet datasets by automatic techniques.

Reference:
1. DEFT: Distilling Entangled Factors by Preventing Information Diffusion
review_title: null

Meta-review: metareview: This paper proposes a method to learn meaningful representations of data by incorporating a pick-odd-one-out task on triplets of images to learn embeddings through variational inference  using a spike-and-slab Gaussian prior.

The reviewers agreed that the paper was well written, had a clear narrative, that the results appeared convincing and that the use of the spike-and-slab prior to determine appropriate dimensionality was novel and interesting.

Where pertinent questions were raised by reviewers on the exposition of the estimation of the upper bound, on the validity of the triplet task and its phrasing, and on the utlity of employing VI over a prior model (SPoSE), the authors provided responses that appear to address these issues reasonably.

The primary issue with the manuscript appears to be mainly with framing. A graphical model, with annotations for the triplet observations, or something similar---a figure to explain the model basically---would have helped make things a bit easier to situate for the reader.

On balance, though it appears the paper has more merits than issues. I would strongly urge the authors to actually make the edits discussing other baselines (Reviewer VDhE), and potentially having the discussion on determining the number of latent dimensions in the main paper rather than the supplement as it appears to be an important distinguishing feature over prior work.

confidence: Certain
Decision: Accept