Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: To approximate human judgements without increasing cost and losing reliability, the authors proposed propose a new class of similarity approximation method based on language, namely captions and stacked words. Their method achieved good approximation with only O(N) computation cost. They also developed the novel overall flow to compute representational similarity. They also claim they are going to make the human judgement dataset publicly available.
strength_and_weaknesses: [Strength]
The concept of the paper is nicely presented for the next AI abundant society where demand for human judgments is more increased. They proposed the novel similarity approximation method with a smaller number of judgements.  
They provide the evaluation data in the supplemental material, which is beneficial.
Overall, I believe what the authors done is quite systematic and very beneficial for the future refinement of the DNN model.

[Weakness]

-	Representational similarity
For beginners, the word of representation similarity is not easy to follow. At least, authors should define the terminology in the introduction.


-	The title
In particular, the linkage between title and abstract seems not clear at the first glance.
Please consider to update the title and abstract so that readers can easily understand the topic theme.


-	Is English best?
There are many languages. In my opinion, I view that authors selected English language to represent the basis of all things in nature. Why English(e.g., language system such as polysemous, grammar, dataset availability…)? Justification of English employment would make the paper better. 

Related to above topic, the paper report “N = 1,492 US participants for the new behavioral experiments reported in this paper.” Are they all Americans and share the same contexts? (e.g., cultures…) 

-	Crowd scouring
How long does it take to STEP-Tag? It is good that measure actual time that participants consumed. Comparing with that with the time used for writing captions may strengthen your claim. 

-	Discussion
I feel strange because the paper end with “Discussion”. Please consider adding conclusion section or rename the last section as “Discussion and Conclusion”.

clarity,_quality,_novelty_and_reproducibility: -	Quality
Good. Issues for human judgment cost have been solved with the proposed idea. 


-	Clarity
Good. In the experiment, model they used is sufficient. 


-	Originality
Good. The idea of methods themselves are not very novel. However, I believe the exhaustive amount of examination is valuable, which increase the originality, thus it should be highly evaluated.

summary_of_the_review: I think this paper should be accepted for the “Human like” evaluation point of view as DNN becomes more sophisticated. I believe their finding is also beneficial for the community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new class of similarity approximation methods based on language. To collect the language data required by these new methods, the authors also developed and validated a new adaptive tag collection pipeline, which is significantly cheaper than the classical methods. Finally, the authors also develop ‘stacked’ methods that combine language embeddings with DNN embeddings, and find that these consistently provide the best approximations for human similarity across all the three modalities. 
strength_and_weaknesses: Strength:
1. The paper is overall well-written and easy to follow.
2. Multiple datasets are evaluated for the proposed method.
3. Multiple modalities are explored and evaluated on the proposed method.

Weakness:
1. The technical contribution of the proposed method is weak.
2. No baselines are compared to the proposed method in the experiment section.
clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written. The authors consider multiple modalities and datasets to evaluate their proposed method. But the novelty of the proposed method is weak. 
summary_of_the_review: The main issue of the paper is the novelty of the proposed method. Based on technical novelty and insufficient experiments, I don’t think the current version meets the standard of the ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors make the observation that representations learned by DL models produce proximity scores very different from human evaluations. They introduce a simple and scalable technique to make the human and model produced similarity scores closer. Essentially, text descriptions or tags for various input data points (across modalities) are passed through LLMs (or word-frequency methods) and the resulting representations used for proximity scores. Stacked representations combining the existing model and the text description representation is shown to consistently match human similarity metrics better.
strength_and_weaknesses: Strengths: I think this is a well-motivated problem. Learned representations are often used as features in the small-data regimen or sometimes directly for getting proximity scores in an AI setting. This paper address the human interpretability of these representations by (i) confirming that human similarities and proximity scores from models can vary a lot, (ii) text-descriptions or tags can be leveraged and stacking these representations with the model-learned representations can help. I also appreciate that the technique is scalable and in many cases not that much of an overhead to implement. I appreciate the arguments in the related text that leverage cognitive science literature. In addition, the paper is easy to follow.

Weaknesses: The paper doesn't have too many weaknesses. I was wondering if we could get some numbers on if the stacked representations help in additional downstream tasks like say classification (i.e. does the performance on imagenet improve if you use imagenet + text). However, I understand that this can be significant undertaking and do not want to base my review on this experiment but it is a potential future direction.
clarity,_quality,_novelty_and_reproducibility: The paper presents a straightforward but well-motivated idea. Having representations match human similarity judgement is indeed useful.

Implementing this technique would require (i) ability to acquire free-text or tag values which are domain-dependent but not prohibitively expensive, (ii) ability to obtain LLM representations which are straightforward since high quality implementations and libraries exist for these now. Reproducibility is not an issue.


summary_of_the_review: - Well motivated problem 
- Clearly described technique that is scalable, easy to implement
- Techniques like these that are easy to implement and help with interpretability are of great use in the small-data regimen (where the bulk of us are). I would like to see this paper at ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors consider the task of approximating human didactic
similarity judgments over N pairs of (images/audio/text). While high
quality human judgments over all pairs is ideal, (N choose 2) is too
big for large N. The authors consider gathering N descriptions of the
objects (tags or captions), extracting text representations of those
descriptions, and then computing text-text similarity. Experiments
across a large number of models and datasets reveal the surprising
effectiveness of computing text-text similarities.
strength_and_weaknesses: Reconstructing pairwise human similarity judgments across N objects is
an interesting task, and the author's idea of gathering textual
descriptions is clever. The performance of the stacked models (which
depend on domain-specific pretrained models and text-text
similarities) and text-only models are surprisingly strong across a
variety of domains --- this finding has implications for the relative
surprising representational strength of text. The annotation process
they describe for their tag corpus is interesting --- multiple rounds
of annotations are undertaken with crowdworkers fixing each-other's
errors. The authors consider a large number of models across many
domains --- the benchmark experiments are extensive, and the "size
vs. accuracy" plots in the end are cool. Figure 5 is amazing --- few
works offer such clear practical guidance.  Finally, I liked the
hopeful message in the end --- that a combined multimodal approach
seems to work best, so don't completely discard the domain-specific
representations.

It should be noted that the cited work where some of the data is from,
Marjieh et al. 2022, is quite similar in the sense that it also
proposes to estimate human similarity judgments. This work appears to
extend that work by exploring more types of models and their
combinations.

My main concerns are that this set of experiments suggests some clear
next setups that I don't nesc. think are out of scope for this work.
Specifically:

- If the goal in practice is to reconstruct human judgments --- I
  would have liked the authors to compare against a supervised setup.
  How much of the remaining misalignments can be fixed by gathering n
  << N^2 additional pairwise judgments and then training a supervised
  model ontop of the models?

- I would have liked to have seen more experiments with even less
  supervision --- is it possible to gather n < N captions, and train a
  model on those very few pairs to map to text, and then use the
  resulting modality --> text model as input to the models shown here?

- I would have appreciated more discussion of tags vs. captions for
  the images. In figure 3A, I assume that the text only methods use
  captions unless specified otherwise. But --- because the tags were
  not fed to, e.g., BERT/RoBERTa, I can't tell if the performance
  gains are due to the full captions being used, or if the gains are
  due to the more expressive models. A trivial linearization of, e.g.,
  "A photo of a tag1, tag2, and tag3" handed to a LLM would have been
  a nice comparison to see.

- Finally, I would have been interested in a fully LLM setup where
  instead of pairwise cosine similarities, both text pairs are fed to
  a very large LLM (e.g., OPT or GPT3) and the model outputs a likert
  rating as text.
clarity,_quality,_novelty_and_reproducibility: As I mentioned, the novelty is somewhat limited because of the prior human
judgements work the authors get their data from. But, the extensive set of
experiments is interesting/new.
summary_of_the_review: Overall, this work address an interesting task/approach ---
approximating pairwise similarity judgments over N objects with only
O(N) annotation via language models. While this setup has been
considered by Marjieh, this work presents more experiments that show
the text representations are surprisingly strong. They consider a very
large number of models, modalities, etc. The released tag sets would
likely be useful to someone. But, a handful of missing additional
experiments, which are arguably in-scope for the present work, are not
presented. And, prior work has already made the observation that LLMs
can approximate similarity judgments across non-textual modalities,
which appears to somewhat limit the novelty of this work.


After response:

The authors seemingly ran all of the experiments I suggested and have incorporated them into the paper. Amazing :-)
Furthermore, the authors clarified the difference with prior work.
All my concerns are addressed, so I raised my score from 5-->8
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper extensively evaluates embeddings of pre-trained deep neural networks in approximating human similarity judgement of image, audio, and video pairs. It empirically shows that embeddings extracted by the language models on tags of samples better approximate human similarity judgements, which can be further improved by stacking embeddings of language models and pre-trained deep neural networks. This is an interesting finding. The paper also introduce new tagged datasets of image, audio, and video pairs. The manuscript is well-written, and the authors did a good job addressing the reviewers' comments in the rebuttal. 
justification_for_why_not_higher_score: The paper extensively evaluates embeddings of pre-trained deep neural networks in approximating human similarity judgement of image, audio, and video pairs. It empirically shows that embeddings extracted by the language models on tags of samples better approximate human similarity judgements, which can be further improved by stacking embeddings of language models and pre-trained deep neural networks. As such, it can be a good poster paper.
justification_for_why_not_lower_score: The paper received good scores from all reviewers, and the authors did a good job in addressing the reviewers' comments.
Decision: Accept: poster