Words are all you need? Language as an approximation for human similarity judgments
Authors: Raja Marjieh
Pol Van Rijn
Ilia Sucholutsky
Theodore Sumers
Harin Lee
Thomas L. Griffiths
Nori Jacoby
Venue: ICLR 2023 poster (2023)
Keywords: cognitive science
language
perception
representational similarity

Abstract: Human similarity judgments are a powerful supervision signal for machine learning applications based on techniques such as contrastive learning, information retrieval, and model alignment, but classical methods for collecting human similarity judgments are too expensive to be used at scale. Recent methods propose using pre-trained deep neural networks (DNNs) to approximate human similarity, but pre-trained DNNs may not be available for certain domains (e.g., medical images, low-resource languages) and their performance in approximating human similarity has not been extensively tested. We conducted an evaluation of 611 pre-trained models across three domains -- images, audio, video -- and found that there is a large gap in performance between human similarity judgments and pre-trained DNNs. To address this gap, we propose a new class of similarity approximation methods based on language. To collect the language data required by these new methods, we also developed and validated a novel adaptive tag collection pipeline. We find that our proposed language-based methods are significantly cheaper, in the number of human judgments, than classical methods, but still improve performance over the DNN-based methods. Finally, we also develop `stacked' methods that combine language embeddings with DNN embeddings, and find that these consistently provide the best approximations for human similarity across all three of our modalities. Based on the results of this comprehensive study, we provide a concise guide for researchers interested in collecting or approximating human similarity data. To accompany this guide, we also release all of the similarity and language data, a total of 206,339 human judgments, that we collected in our experiments, along with a detailed breakdown of all modeling results.

Published as a conference paper at ICLR 2023
WORDS ARE ALL YOU NEED? LANGUAGE AS AN APPROXIMATION FOR HUMAN SIMILARITY JUDGMENTS
Raja Marjieh',, Pol van Rijn", Ilia Sucholutsky", Theodore R. Sumers', Harin Lee²A,
Thomas L. Griffiths! Nori Jacoby2."
Equal contribution.
'Department of Psychology. Princeton University
Max Planck Institute for Empirical Aesthetics
Department of Computer Science, Princeton University Max Planck Institute for Cognitive and Brain Sciences
ABSTRACT
Human similarity judgments are powerful supervision signal for machine learn- ing applications based on techniques such as contrastive learning, information retrieval, and model alignment, but classical methods for collecting human similar- ity judgments are too expensive to be used at scale. Recent methods propose using pre pre-trained deep neural networks (DNNs) to approximate human similarity, but pre-trained DNNs may not be available for certain domains (c.g., medical images, low-resource languages) and their performance in approximating human similarity has not been extensively tested. We conducted an evaluation of 611 pre-trained models across three domains images, audio, video and found that there is a large gap in performance between human similarity judgments and pre-trained DNNs. To address this gap. we propose a new class of similarity approximation methods based on language. To collect the language data required by these new methods, we also developed and validated novel adaptive tag collection pipeline. We find that our proposed language-based methods are significantly cheaper, in the number of human judgments, than classical methods, but still improve performance over the DNN-based methods. Finally, also develop 'stacked' methods that combine language embeddings with DNN embeddings, and find that these consis- tently provide the best approximations for human similarity across all three of our modalities. Based on the results of this comprehensive study, we provide a concise guide for researchers interested in collecting or approximating human similarity data. To accompany this guide, we also release all of the similarity and language data, a total of 206,339 human judgments, that collected in our experiments, along with detailed breakdown of all modeling results.
INTRODUCTION
Similarity judgments have long been used as a tool for studying human representations, both in
cognitive science (Shepard 1980 1987 Tversky 1977: Tenenbaum & Griffiths 2001), as well as in
neuroscience, as exemplified by the rich literature on representational similarity between humans
and machines (Schrimpf et al., 2020 Kell et al. 2018 Linsley et al. 2017] Langlois et al. 1021
Yamins et al. 2014) whereby similarity patterns of brain activity are compared to those arising from model of interest. Recent research in machine learning suggests that incorporating human similarity judgments in model training can play an important role in a variety of paradigms such as
human alignment (Esling et al. 2018). contrastive learning (Khosla et al. 2020), information retrieval (aarkkh et al. 2020), and natural language processing (Gao et al. 2021..
However, building large dataset based on human similarity judgments very expensive and often infeasible since the number of judgments required is quadratic in the number of stimuli - for N
"Correspondence: (raja. vraja.marjiio marjieh, is2961 aprinceton. edu, pol van-ri waddeae..pg....

---

Published as a conference paper at ICLR 2023
How similiar are the animals?
Human ground truth
O(N?)
human interactions
Painwise similarity
Multimodal datasets
Prediction approaches
audio
images
video
human ((1) interactions
DNN similarity
human O(N)
Stacked
interactions
cc Captions
LLM similarity
O(N)
human interactions
+
Tags
WFA similarity
Comparing human similarity scores gathered through crowdsoureing with ML pipelines. We used data from three modalities: images, audio, and video. For each modality, we extracted deep model embeddings and gathered human captions and tags. Word- and language- embedding models, as well simple word-frequency analysis, were used to predict human similarity judgments.
Figure 1:
stimuli, O(N3) judgments are required") For example, to fully quantify the similarity of all possible dyadic pairs of 50,000 images, one needs to collect on the order of 1.25 billion (~ 50000" human similarity judgments. Thus, human judgments are the main bottleneck for machine-learming methods based on similarity. For this reason, the majority of available human similarity datasets small by machine learning standards (up to few thousand objects).
Advancements in deep learning have brought alternative approach that does not require extensive collection human judgments. Specifically, the idea to use the similarity between hidden repre- sentations in pre-traiiee are-trained deep neural networks (DNNs) to approximate human similarity (Peterson
et al.) 2018 Tha et al. 2020, Marjieh et al. 2022 Hebart et al., 22000 Roads Love 2021). Some of these methods also suggest fine-tuning representations
small training set of human similarity judgments (Peterson et al. 2018). This, in turn, results in a significant reduction in the number of required human judgments down to (11) (given the pre-trained model). While such methods are promising, they still require access to strong pre-trained models which may not necessarily be available all domains (e.g., medical datasets, niche modalities, low-resource languages, etc.). In addition, representations obtained from neural networks may not always overlap with human similarity representations, given that the models can be trained for different objectives (i.e., their embeddings may be poor approximations for human similarity).
comprehensive comparison to assess which models perform well in predicting human similarity across different modalities is currently lacking in the literature. To this end, one of our main contributions in this paper is providing first-of-its-kine large-scale evaluation of over 600 publicly- available pre-traine pre-trained models as approximations for human similarity judgments three modalities
Depending on various assumptions, the full range of classical methods can require between O(N log N) Jamieson & Nowak 2011) and ((N3) (Hebart et al. 2000)) human judgments. In this work, used O(N2) human judgments (collecting all unique dyadic pairs) the baseline for comparison
2

---

Published as a conference paper at ICLR 2023
(images, audio, video). Our experiments reveal that there is large gap in performance between the ((1) DNN methods and the classical ((N2) similarity method we used as the baseline.
To address this gap, propose new class of ((N) methods to efficiently and accurately ap- proximate human similarity based on language. This is motivated by a long line research in cognitive science suggesting that language
extremely efficient way for humans to communicate information about their sensory environment (Murphy 2004; Zaslavsky et al. 2018 Piantadosi et al.) 2011, Jaeger & Levy 2006). This in turn suggests that can use textual descriptors approximate similarity judgments across different modalities. Moreover, such textual descriptors can be collected the cost of O(N) human judgments (as people describe individual stimuli rather than pairs), which renders this method scalable.
We consider two approaches for approximating similarity from text data. One approach to use pre-trained Large Language Models (LLM) to produce vector embeddings of the textual descriptions, and then use a measure of distance between these embeddings to approximate human similarity. This method is more domain-agnostic than the ((1) deep learning methods it only requires access to pre-trained LLM regardless of the modality the original dataset. However, there are some cases where the domain may be out-of-disrrbbution for all available LLMs (e.g., niche technical fields), or where no LLMs are available at all (c.g., low-resource languages). In such cases, the other approach to use Word- Fredu-Fryyyncyy Analysis (WFA) methods from classical text processing literature (Barrios
et al. 2016 Rouge 2004
Beel et al. 2016).
As for the textual descriptions themselves, consider two types, namely, free-text captions and concise word tags. Collecting captions for machine learning datasets is a well-established practice and can easily be done through crowdsourcing platforms. On the other hand, there is consensus on best practices for collecting tags without a pre-existing taxonomy (i.e., open-set labels). To address this, we propose a novel adaptive tag mining pipeline called Sequential Transmission Evaluation Pipeline (STEP- Tag)) which we describe in Section [.2.4] As we will show, STEP-Tag allows to collect meaningful, diverse, and high-quality word tags for target stimuli in an online crowdsoureing environment.
Finally, we propose one additional set of hybrid approximation methods that combine sensory information with textual descriptions while still requiring O(N) human judgments. For this approach, we propose to stack the embeddings derived from both domain-specific models (e.g., output from the last layer of an image classifier) with the LLM embedding of the respective textual description. When multi-modal models are available, we can similarly leverage the joint embedding of both the stimulus and its textual description.
We evaluate all of these novel and existing methods across multiple modalities. We test the relative contributions of linguistic and sensory information in approximating human similarity and show that our proposed language- based methods provide both accurate and efficient approximations across modalities, Crucially, with this large-scale evaluation, we are able for the first time to provide researchers with a comprehensive guide of the tools to for approximating human similarity scale.
though they do not require trained modality-specifie deep learning model.
To summarize, our contributions are as follows:
We conduct a comprehensive comparison of human similarity approximation methods.
We propose a novel modality-agnosti method for approximating similarity based on text and show that it is both efficient and competitive in terms of performance.
We propose STEP-Tag, a novel adaptive tagging pipeline, and show that it is effective for crowdsoureing high-quality and diverse sets of word tags.
We synthesize our findings into detailed guide for researchers interested in approximating human similarity judgments at scale.
We collect and release ground-truth and approximated versions of a large behavioral dataset (N = 1,492) across three different domains (images, audio, video), including two text- approximated similarity matrices for 1,000 audio clips and 1,000 video clips.
3

---

Published as a conference paper at ICLR 2023
2 DATASETS
2.1 STIMULI
Throughout this work, we considered five stimulus datasets across three different modalities images, audio, and video consisting of total of 31,320 dyadic pairs labeled with similarity.
Images For images, considered three datasets of common objects introduced in Peterson et al. (2018) - namely, animals, furniture, and vegetables each consisting of 7,140 dyadic pairs (all unique pairs over 120 images).
Audio For audio, we used the RAVDESS corpus [Livingstone & Russo (2018), released under CC Attribution license), which consists of semantically neutral sentences spoken by 24 US American actors convey specific target emotion. To construct 1.000reccrdinn subset, selected emotions per speaker per sentence. We randomly omitted 104 emotional stimuli and included all 96 neutral recordings (the dataset only contains 2 neutral recordings per speaker per sentence). To construct the subset composed of 4,950 dyadic pairs (all unique pairs over 100 recordings), we randomly selected ~3 recordings per emotion from the 1,000.
Video Finally, for the video dataset, considered the Mini- Sommi-Knnetics222 220 dataset (Xie al. 2018) (released under a CC BY 4.0 International License), which contains a large set of short video clips of human activities from 200 activity classes. Specifically, contains 5,000 videos in total. To construct our 1, 1,000-video dataset, we sampled random videos from each of the 200 activity categories. The 100-video subset (4,950 dyadic pairs) used in the similarity judgment collection experiment was then generated by sampling 100 random stimuli from the 1,000 list.
focused the validation split, which
2.2
HUMAN JUDGMENT COLLECTION
2.2.1 PARTICIPANTS
We collected data from N 1,492 US participants for the new behavioral experiments reported in this paper. Participants were recruited anonymously from Amazon Mechanical Turk and provided informed consent under an approved protocol by either the Institutional Review Board (IRB) at Princeton University (application 10859) or the Max Planck Ethics Council (application 2021 42) before taking part. Participants earned 9-12 USD per hour, and each session lasted less than 30 minutes. To help recruit reliable participants, reside in the United States and have participated in more than 5,000 previous tasks with a 99% approval rate (see Supplementary Section for additional details about the behavioral experiments). All experiments were implemented with the Dallinger and PsyNet frameworks designed for automation of large-scale behavioral research (Harrison al. 2020). Supplementary Section A. include the data that was collected, instructions used, and code for replication of the behavioral experiments. We also provide the code for computational experiments and analysis.
required that participants
least 18 years of age,
2.2.2 SIMILARITY JUDGMENTS
We collected two batches of pairwise similarity judgements, one for each of the audio and video subsets, and were provided access the similarity matrices for the three image datasets by the authors Petersoo et al. (2018). For each pair we collected ~ 5 similarity judgments to average out inter-rater noise.
2.2.3 CAPTIONS
We collected free-text captions for the video and audio datasets. Captions for the image datasets were already collected by Marjich et al. (2022) and used here with permission. For each stimulus, we collected ~ 10 captions.
4

---

Published as a conference paper at ICLR 2023
Iteration 3
Iteration 2
tomatillo
Iteration
tomatillo
tomatillo
brussel sprouts
Iteration
Iteration 4
green
tomatillo
brussel sprouts
brussel sprouts
*
*
Figure 2: STEP-Tag, our novel tag-mining paradigm. We adaptive process in which results of one iteration are used as inputs for subsequent iterations. In every iteration, participants can add new tag, rate the relevance of existing tags or flag tags that are inappropriate.
2.2.4 TAGS
We propose a novel adaptive tag pipeline for simultaneous data collection and evaluation called Sequential Transmission Evaluation Pipeline (STEP) and apply it in the context of semantic tag mining (STEP-Tag). Our paradigm, STEP-Tag, allows researchers to efficiently collect high-quality word tags for a given stimulus (Figure 2) and extends existing crowdsourcing text-mining techniques (Von Ahn
& Dabbish 2008 2004 Krishna al.) 2017 Law al. 2007) by integrating ideas from transmission
chain experiments (irbyy et al. 2008; Griffiths & Kalish 2005). In STEP-Tag, participants adaptively
create tags for set of target stimuli and simultaneously evaluate the annotations made by previous participants. In each trial, participants are first given a stimulus (e.g., an image or audio fragment) and rate the relevance of tags that were created by other participants (on a 5-interval Likert scale) or flag tag if they find it inappropriate (with tags removed if more than two people flag the tag). Next, participants are also given the opportunity describes the stimulus is missing. The results of the annotation procedure of one participant then propagate to the next participant (additional details about the paradigm, and screenshots are provided in Supplementary Section (B.6). Ultimately, as the process unfolds over many iterations, meaningful tags are extracted and validated by multiple participants, enabling efficient open-label collection of desired dataset.
add new tags if they feel a relevant tag that
To validate STEP-Tag, we compared it against several baselines: (i) randomly selecting only a single high-rated tag from the last iteration of STEP-Tag per stimulus, (ii) using tags only from the first iteration of STEP-Tag (equivalent to non-adaptive tag collection), and (iii) using class labels instead of tags. We found that tags produced after multiple iterations of STEP-Te Tag outperformed all three baselines in terms of quality (i.c., downstream performance for similarity reconstruction) and diversity (see Supplementary Section B.6.
MODELS
3.1 DNN-RASED METHODS
We tested a wide range of pre-trained ML models that do not rely on text (overall we tested 61 I models) and compared their internal representations to human similarity judgments and text-based predictions (Figure [TA). We compiled our model pool by leveraging pre-trained model reposito- ries (or zoos) available online. In particular, for images we use 569 pre-trained models from the wttorch-image -odgeess package timm (Wightman] 2019), for audio use 36 pre-trained models available in the torchaudio o package (Yang et al. 2021) (see also Supplementary Figure pre-trainne trained models available from the
10 for an analysis of layer depth), and for video PyTorchVides package (Fan et al. [202T). Because of the recent success of multimodal training, additionally included multimodal models based on CLIP from OpenAl's public implementa-
5

---

Published as a conference paper at ICLR 2023
tion
for the image datasets, and compared them to "stacked" representations (i.e., concatenating embeddings from separate image and text models).
3.2
BLSED METHODS
Tags embed tags we used ConceptNet Numberbatch (CNNB) which a word-embeddingg model trained on the ConceptNet knowledge graph that leverages other popular word embedding models such as word2vec and GloVe (Speer et al. 2017). We experimented with several algorithms for computing similarity between sets (or multi-sets) of tags and share the details in Supplementary Section C.1.2 As control, for images we also tried converting tags into a caption the form "This is an image of tagl, tag2. ... and embedding them using a language model (see Supplementary Section C.I.2).
Captions To embed captions, we used four pre-trained LLMs from HuggingFace (Wolf et al. 2020): -bert-basc-uncased 'deberta-Xe 'deberta-xlarge-men xlarge-mnli', 'sup-simese- hettt/ss--beet-bb---nnsssseeeeee and 'sup-simese- roberta-large'. SimCSE is pre-training procedure that uses semantic entailment contrastive learning objective (Gao al. 2021). According BERTScore (Zhang et al. (2200), the latter three models are ranked in the top 40 models in terms of correlation with human evaluations on certain tasks, with beeeera-xlarge-mnll ranked first. However, in our experiments, we found that embedding similarity computed from sun-simcsee-oobett--aaa judgments out of the four models. For SimCSE- models. we used representations from the (final) embedding layer (where the SimCSE contrastive objective is actually applied). For the other two models, we computed embeddings from every layer, but restricted the main analysis to embeddings from the penultimate layers. This was done in order to be consistent with our procedure for DNNs.
has the highest correlation with human similarity
Other methods For the image datasets, we also considered several other methods that made use of LLMs but do not fit into the categories described above. One approach was using prompts with GPT3 (Brown et al. 2020pp in a text-completion setup to directly predict similarity without extracting embeddings (see Supplementary Section C.1.3 for details). We also tried using pre-trained image captioning models to generate captions automatically (i.e. this would reduce O(N) language- baasa methods to (1)) but this resulted in poor performance (see Supplementary Section C.1.3 for details).
3.3 STACKING METHODS
We produce stacked representations for each modality by concatenating the single best-performing (see Figure B) LLM's ' embeddings with the embeddings from the five best pest-performig DNNs into single set long embeddings. Since the two sets of embeddings come from different spaces, we add single tunable hyperparameter for rescaling the LLM embeddings. This hyperparameter can be set manually, but we use a small number of ground-trui ttuth similarity judgments (we use dyadic pairs for just 20 stimuli) to optimize it automatically.
3.4 WORD FREQUENCY ANALYSIS (WFA) METHODS
The aim of the WFA methods is to enable similarity approximation from language using traditional embedding techniques. Such techniques are particularly useful for low-resource languages or cross-cultural comparisons (Cowen & Keltner 2017] (Barrett 2020). for which pre-trained models are lacking, as they work solely on the basis the text itself. The WFA methods we considered included measuring co-occurrence, Rouge score, bm25s, and tf-idf. We provide details each of these procedures in Supplementary Section
3.5 PERFORMANCE METRIC
We quantified performance by computing the Pearson correlation r between approximated similarity scores and the ground-truth human similarity scores for all the unique dyadic pairs in dataset. We compared the performance the different prediction methods to the inter-rater reliability (IRR) of participants, which serves approximate hoper-bound on performance. Following Peterson et al.) (2018), we computed IRR for each human similarity matrix using the split-half correlation method with Baaarmnn-Broww correction (Brown) 1910).

---

Published as a conference paper at ICLR 2023
4 RESULTS
A
0.9
Inter-rater reliability
0.8
Mean correlation
0.7
0.6
0.5
0.4
0.3
Tap www.mraae.eee
ePt.)mmatt
0.1
0.2
LAMAT
CUP
3001
Cia
Cirlit
Bel
Tapitralit
Thine
Tapysur
Backete
Celira
0.9
0.8
Inter-rater reliability
Mean correlation
0.6
0.5
0.4
0.3
0.2
Caplore m0hee
0.1
0.9
Inter-rater reliability
0.8
Mean correlation
0.7
0.6
0.5
0.4
0.3
0.2
0.1
D
Images
Audio
Video
0.6
0.8
0.6
0.8
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
Approach
WFA
DNN (baseline)
LLM
Stacked
Figure 3: Correlation human similarity. A: Top 50 models averaged the 3 image datasets. B: Audio dataset. C: Video dataset. Each DNN baseline bar averages over multiple variants of the same architecture; the dots indicate average correlation individual variants of the architecture. D: Average for each method type for each modality. The error bars are standard deviations.
Figure 3 summarizes the performance the various techniques across the three modalities. Note that the image modality results in Figure J are averaged across the three image datasets and only show the top 50 methods for this modality due to space constraints. Figure 3D shows the mean performance of the methods of each type for each modality. When viewing these results, clear hierarchy emerges. While no approximation methods can perfectly match the ground-true truth pairwise similarity, (see the gap between the methods and IRR), stacked ones get close and consistently more aligned with
7

---

Published as a conference paper at ICLR 2023
human similarity than other methods across all three modalities. Text-based methods come next in this hierarchy, followed by DNN- based ones. We also considered supervised methods that reweight DNN-based embeddings based on a small set of human similarity judgments, but we found that the performance was unstable (see Supplementary Section
C.3 for details).
The pre-eminence of stacked results suggests that LLMS and DNNs capture at least some different sources of variance in human similarity judgments. This is reinforced by our surprising finding that stacked representations from CLIP, sfate-of-the-art jointly pre-trained multi-modal model, do not outperform stacked representations from independently trained models. We hypothesize that this happens because information is lost from both modalities when optimizing for a joint embedding. However, we note that the modest size of the performance gap between stacked and LLMs/DNNs, suggests that there also significant overlap between aspects of human similarity captured by language and perception.
To investigate the effect of architecture and downstream task (e.g.. classification) performance on alignment of DNNs with human similarity, for the image modality we compared similarity approximation performance against the number model parameters on a log scale (Figure[AA) and ImageNet classification performance (Deng et al.) 2009) (Figure Overall, we found positive correlation between similarity approximation performance and the number of model parameters (r 0.39, < 0.001) and a smaller but still significant positive correlation with performance on ImageNet (r = 0.26, p < 0.001), There were some notable exceptions with particularly high ImageNet performance but low similarity performance, such as the image transformer BEiT (Bao ct al. 2021
Finally, we leverage both DNN-based methods and our proposed language-based methods to approxi- mate similarity matrices that would otherwise require an unaffordable number of human similarity judgments collect all dyadic pairs. Specifically, we approximate the two similarity matrices corresponding to all 1,000 audio clips and 1,000 video clips in our datasets using every method listed for each of those modalities in Figure 3 We provide visualizations of the resulting matrices at fihttpss w/w////r-s-oo///o///////e are aaree-lll--oo-eeeee need. s3 amazonaws com/ index. .ttll We note that to exhaustively collect all dyadic pairs with five judgments per pair would normally require roughly 2.5 million human judgments for each of these matrices.
A
Correetton with human simiterity
coaine
0.8
04+
JCT
0.-
Emig
Pachaal Moler
Be Straree
OCT
10
1ex08
Number parameters
Correlstion human similaitty
0.6
Pariy
Pill
04-
02-
dieetes
70
80
80
ImageNel accuracy
Figure 4: Correlation to human similarity judgments as function of A: number of model parameters; and B: ImageNet accuracy.
8

---

Published as a conference paper at ICLR 2023
How large is my dataset?
Start
N=1000
N-1000
Can I collect text descriptions?
Collect human similarity judgments O(N2)
Pick a subset of your data
no
yes
no
access
have access pre-trained DNN?
pro-trained LLM?
yes
no
Does my LLM embed captions or tags?
Collect text and use frequency analysis O(N)
Use the similarity between DNN embeddings O(1)
captions
tags
no
Use the similarity between embeddings O(N)
Collect captions
Collect tags with STEP-Tag
cC
Use the similarity between stacked embeddings O(N)
yes
Do have access to a pre-trained DNN?
Figure 5: Guide to collecting and estimating human similarity judgments at scale.
5 DISCUSSION AND CONCLUSION
In this work, we compared novel and existing methods for approximating human similarity judgments. The main contributions can be summarized as follows: 1) we provide a simple and accessible approach for approximating O 2 human similarity judgments using O(N) annotations, 2) we propose a new adaptive pipeline STEP-tag for tag mining, 3) we evaluate our approach against 600+ domainn-speciff
state-of-thc-art DNNs, and 4) we publicly release all data comprising 206,339 human judgments.
Based on these, we are now able provide researchers with a best-practices guide to collecting similarity datasets. Our guide is based on two bottlenecks that researchers may face: one is the limit on the number of judgments that can be collected (e.g., due to cost) and the second is the availability of pre-trained models (i.e., either DNNs or LLMs). Our results make it clear that deep learning can provide good approximations for human similarity. In fact, when both pre-trained L.LMs and DNNs available, stacking their representations is consistently the best approach. However, neither type of pre-trained models are available, we suggest that classical word- cord-frequency analysis methods still provide researchers with an efficient and competitive method for approximating human similarity. Our guide, comprehensively covering these and other cases, is laid out in Figure
when
One limitation of this work is that while similarity proxies generated from our pipeline can support ML datasets, they are also at risk of baking in high-level human biases that can lead to adverse societal implications, such as amplifying race and gender gaps. Researchers should devote utmost care to what they choose to incorporate in their training objective. Another limitation of our work is the fact that we were restricted to English text data and US participants. However, we believe that our approach and proposed methods (especially STEP-tag and the word- pord-frequency methods) pave the way for the study of cross-cultural variation of human semantic representations by providing efficient tools for crowdsourcing high-quality semantic descriptors across languages. This is particularly relevant for low-resource languages, where our tag- -aimmining techniques can work even with the absence of pre-trained ML models (Thompson et al. 2020 Barrett 2020), We are currently expanding our work to include more languages and diverse cultures. Taken together, our results showcase how we can leverage language to make machine representations more human-like. Moreover, it highlights the importance of combining machine learning and cognitive science approaches for mutually advancing both fields. In particular, believe that the methodologies adopted in this work have the potential to greatly advance basic research on naturalistic representations in cognitive science.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by grant from the John Templeton Foundation to TLG, an NDSEG fellowship to TRS, and NSERC fellowship (567554-2022) IS.
REFERENCES
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for supe-vuperei learning speech representations. Advances in Neural Information Processing Systems, 33:12449-113 2020.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: general framework for self-supervised learning in speech, vision and language, 2022.
Hangbo Bao, Li Dong, and Furu Wei, BEiT: BERT pre-training of image transformers. arXiv preprint arXiv:2/06.00 08254, 2021.
Clark Barrett. Towards cognitive science of the human: cross-cultural approaches and their urgency. Trends in Cognitive Sciences, 24(8):620-638, 2020.
Federico Barrios, Federico López, Luis Argerich, and Rosa Wachenchauzer. Variations of the similarity function of textrank for automated summarization. arXiv preprint arXiv: arXiv: 1602.03666 2016.
Joeran Beel, Bela Gipp, Stefan Langer, and Corinna Breitinger. Paper recommender systems: literature survey. International Journal on Digital Libraries, 17(4):305-333 -388, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
William Brown. Some experimental results the correlation of mental abilities British Journal of Psychology. /904-1920, 3(3):266-32,, 1910.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. WavLM: Large-scale self-supervised pre-training for full stack speech processing, arXiv preprint arXiv:2110. 13900, 2021.
Alan S Cowen and Dacher Keltner. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the National Academy of Sciences, 119033))77000-E7999 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp.
Ieee, 2009.
Philippe Esling, Adrien Bitton, al. Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics. arXiv preprint arXiv:1805.08500 2018.
Haoqi Fan, Tullie Murrell, Heng Wang, Kalyan Vasudev Alwala, Yanghao Li, Yilei Li, Bo Xiong, Nikhila Ravi, Meng Li, Haichuan Yang, Jitendra Malik, Ross Girshick, Mau Feiszli, Aaron Adcock, Wan- Yen Lo, and Christoph Feichtenhofer. PyTorch A deep learning library for video understanding. In Proceedings of the 29th ACM International Conference Multimedia, 2021. httpps://pytorccavddeo.oo org/
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 203-213, 2020.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6202-6211, 2019.
10

---

Published as a conference paper at ICLR 2023
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.0882. 2021.
Thomas L Griffiths and Michael L Kalish. A bayesian view of language evolution by iterated learning. In Proceedings of the Anmual Meeting of the Cognitive Science Society, volume 27, 2005.
Peter Harrison, Raja Marjieh, Federico Adolfi, Pol van Rijn, Manuel Anglada- Tort, Ofer Tcherni- chovski, Pauline Larrouy-Maestre and Nori Jacoby. Gibbs sampling with people. Advances in Neural Information Processing Systems, 33:133::111:: 16659-1066,,, 2020.
Martin N Hebart, Charles Y Zheng, Francisco Pereira, and Chris Baker. Revealing the multidimen- sional mental representations natural objects underlying human similarity judgements. Nature Human Behaviour, 4(11): 2020.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and AbdeIrahman Mohamed. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:3451-3466 2021.
T Jaeger and Roger Levy. Speakers optimize information density through syntactic reduction. Advances in Neural Information Processing Systems, 19, 2006.
Kevin G Jamieson and Robert D) Nowak. Low-dimensional embedding using adaptively selected ordinal data. 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 1077-1084. IEEE, 2011.
Aditi Jha, Joshua Peterson, and Thomas Griffiths. Extracting low-dimensional psychological representations from convolutional neural networks. arXiv preprint arXiv:2005. 2020.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The Kinetics human action video dataset. arXiv preprint arXiv:
2017.
Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Vorman-Haigners and Josh H McDermott, A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, 98(3):630 644, 2018
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33: 1811186611877 2020.
Simon Kirby, Hannah Cornish, and Kenny Smith. Cumulative cultural evolution in the laboratory: An experimental approach to the origins structure in human language. Proceedings of the National Academy of Sciences, (0531))11081-10688 2008
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73, 2017.
Thomas Langlois, Haicheng Zhao, Erin Grant, Ishita Dasgupta, Tom Griffiths, and Nori Jacoby. Passive attention in artificial neural networks predicts human visual selectivity. Advances in Neural Information Processing Systems, 34, 2021.
Edith LM Law, Luis Von Ahn, Roger B Dannenberg, and Mike Crawford. TagATune: A game for music and sound annotation. In ISMIR, volume 3, pp. 2007.
Kristin Lemhöfer and Mirjam Broersma. Introducing lextale: quick and valid lexical test for advanced learners of english. Behavior research methods, 44(2):325 32333333-33,,0 2012.
Drew Linsley, Sven Eberhardt, Tarun Sharma, Pankaj Gupta, and Thomas Serre. What are the visual features underlying human versus machine vision? In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 2706-2714, 2017.

---

Published as a conference paper at ICLR 2023
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Viston, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. arXiv preprint arXi::2201.05444 033545 2022.
Steven R Livingstone and Frank A Russo. The Ryerson audio-visual database of emotional speech and song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)e0016331, 2018.
Raja Marjich, Ilia Sucholutsky, Theodore R Sumers, Nori Jacoby, and Thomas L Griffiths. Predicting human similarity judgments using large language models. arXiv preprint arXiv:2202.04728, 2022.
Alice E Milne, Roberta Bianco, Katarina C Poole, Sijia Zhao, Andrew J Oxenham, Alexander Billig, and Maria Chait. An online headphone screening test based on dichotic pitch. Behavior Research Methods, 53(4): 2021.
Gregory Murphy. The big book of concepts. MIT press, 2004.
Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Waters, and Yinfei Yang. Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for MS-COCO. arXiv preprint arXiv:2004. 15020, 2020.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Joshua C Peterson, Joshua T Abbott, and Thomas L Griffiths. Evaluating (and improving) the correspondence between deep neural networks and human representations. Cognitive Science, 42 (8):2648-22) 2669, 2018.
Steven T Piantadosi, Harry Tily, and Edward Gibson. Word lengths are optimized for efficient communication. Proceedings of the National Academy of Sciences, 108(9):352-3329, 2011.
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chich Chou, Sung- Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. SpeechBrain: A general-purpose speech toolkit, 2021. arXiv:2106.04622
Brett D Roads and Bradley C Love. Enriching ImageNet with human similarity judgments and psychological embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3547-3557, 2021.
Lin CY Rouge. A package for automatic evaluation of summaries. In Proceedings of Workshop Text Summarization of ACL, Spain, 2004.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj. Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Rac Roy, Franziska Geiger, al. Brain-Score Which artificial neural network for object recognition is most brain-like? BioRxiv, pp. 407007, 2020. 2020.
Roger N Shepard. Multidimensional scaling, tree-fitting, and clustering. Science, 2 21064468):390-398, 1980.
Roger N Shepard. Toward a universal law of generalization for psychological science. Science, 237 (4820)::317-1323, 1987.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-first AAAI Conference on Artificial Intelligence, 2017.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pp. 6105-61 14, PMLR, 2019.
12

---

Published as a conference paper at ICLR 2023
Joshua B Tenenbaum and Thomas L Griffiths. Generalization, similarity, and bayesian inference. Behavioral and brain sciences, 24(4):629-66) 640, 2001.
Bill Thompson, Seán G Roberts, and Gary Lupyan. Cultural influences on word meanings revealed through large-scale semantic alignment. Nature Human Behaviour, 4(10): 1029-1038, 2020.
Amos Tversky, Features of similarity. Psychological review, 84(4):327, 1977.
Luis Von Ahn and Laura Dabbish. Labeling images with computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 319-326, 2004.
Luis Von Ahn and Laura Dabbish. Designing games with purpose. Communications of the ACM, 51(8):58-67, 2008.
Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, and Björn W. Schuller. Dawn of the transformer era in speech emotion recognition:
closing the valence gap. 2022.
Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wac Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung yi Lee. SUPERB: Speech Processing Universal PERformance Benchmark. In Proc. Interspeech 2021, pp. 1191-1999999999 2021. doi: 10.24455/mnexxeechh22211111
Ross Wightman. PyTorch image models.
https: https://github: com/ ww.om/rwigoo wwt.an/
pytorch--mage-menn
2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac. Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-at art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language System Demonstrations, pp. 38-45. Association for Computational Linguistics, 2020.
Processing:
Kevin JP Woods, Max Siegel, James Traer, and Josh H McDermott. Headphone screening to facilitate web-based auditory experiments. Attention, Perception, & Psyyhophysics, 79(7); 2064 2072, 2017.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 305-321, 2018.
Daniel Yamins. An optimization- based approach to understanding sensory systems. The Cognitive Neurosciences, 4(V ()33), 2020.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, (2)):8619--622)) 2014.
Yao-Yuan Yang. Moto Hira, Zhaoheng Ni, Anjali Chourdia, Artyom Astafurov, Caroline Chen, Ching-Feng Yeh, Christian Puhrsch, David Pollack, Dmitriy Genzel, Donny Greenberg, Edward Z. Yang, Jason Lian, Jay Mahadeokar, Jeff Hwang, Ji Chen, Peter Goldsborough, Prabhat Roy, Sean Narenthiran, Shinji Watanabe, Soumith Chintala, Vincent Quenneville-Béie and Yangyang Shi. Torchaudio: Building blocks for audio and speech processing. arXiv preprint arXiv:2 15018, 2021.
Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences, 115(31):7337-744. 2018.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https://opente: net/ wow.mmid--Seeuccye....
13

---

Published as a conference paper at ICLR 2023
SUPPLEMENTARY MATERIALS
A STIMULI AND DATA
A.1 CODE AND DATA AVAILABILITY
link is provided to the public, containing all the data collected for this project during the review stage It includes the new human behavioral data, the computational experiments with machine learning models, and all the necessary analyses scripts for producing the results. Additionally, the repository includes the Dallinger/ssyNet source codes for reproducing the behavioral experiments. Finally, we present an interactive visualization for exploring the similarity between stimuli as experienced by humans and different methods reported in the paper
B
BEHAVIORAL PARADIGMS
B.1 PARTICIPANTS
The exact number of participants for each of the 9 new behavioral experiments is reported in Table [
Table I: Behavioral experiment summary table.
Modality
Paradigm
Respect
Total stimuli
Trials per participant
Section
Pre-screening
Images
Tags
Animals
120
60
2.2.4
56
LX
Images
Tags
Furniture
120
60
2.2.4
58
LX
Images
Tags
Vegetables
120
2.2.4
57
LX
Audio
Similarity
Emotions
100
85
2.2.2
252
HT
Audio
Captions
Emotions
1,000
2.2.3
151
HT. LX
Audio
Tags
Emotions
1,000
50
2.2.4
217
HT. LX
Video
Similarity
Activities
100
2.2.2
284
HT
Video
Captions
Activities
1,000
50
2.2.3
196
HT. LX
Video
Tags
Activities
1,000
50
2.2.4
221
HT, LX
Note. 'N' denotes the number of participants included in the analysis; 'LX' denotes the Lex English proficiency pre-screening task; 'HTT denotes the headphone test.
B.2 IMPLEMENTATION
All behavioral experiments were implemented using the Dallinge PsyNet (Harrison et al. 2020) frameworks. Dallinger is a modern tool for experiment hosting and deployment which automates the process of participant recruitment and compensation by integrating cloud-based services such as Heroky with online crowd-sourcing platforms such as AMT. PsyNet a novel experiment design framework that builds on Dallinger and allows for flexible specification of experiment timelines as well as providing support for wide array of tasks across different modalities (visual, auditory and audio-visual). Participants interact with the experiment through their web-browser, which in turn communicates with a backend Python server responsible for the experiment logic.
B.3 PRE-SCREENING
common technique for filtering out participants that are likely to deliver low-quality responses, as well as automated scripts (bots), is to implement pre-screening tasks prior to the main part of
Code and data: httpd////ssseeee//e////////////a
Interactive
plots: htte:///rrdsaareeelggoooeeeee...
index
htm
https: www. hww.ku.eokkkoooo com/
14

---

Published as a conference paper at ICLR 2023
each experiment. Failing the pre-screening tasks results in early termination of the experiment. Nevertheless, participants are still compensated for their time regardless whether they fail or succeed pre-screener to ensure fair compensation. The role of screenesss in our studies was to realize two main criteria for data quality, namely, a) to be able to collect high-quality text deseriptors, and b) ensure that participants component in prosody and videos). To do this, we implemented two pre-screening tasks, an English proficiency and standardized headphone test (used only for audio and video experiments). Table provides details on which pre- screeners were used in each of the behavioral experiments.
able inspect the target stimuli properly (in particular the audio
alberation
Does this word exist?
Figure 6: Example trial from the LexTALE pre-screening task (Lemhöter & Broersma 2012).
English proficiency test. To test participants' English proficiency, we used LexTALE, a lexical decision task developed in Lemhöfer & Broersma (2012). In each trial, participants were briefly presented (1 second) with either a real English word or a made up word that does not exist. Participants were instructed to guess whether the word was real or not. A total of 12 trials (half of them being real words) were presented, and 8 of them needed to be correct for the participant to pass. The presented words were: hasty, fray, stoutly, moonlit, scornful, unkempt, mensible, kilp, plaintively, crumper, plaudate, alberation. An example trial is shown in Figure
Which sound was softest (quietest) -- 2, or 3?
Figure 7: Example trial from the headphone pre-screening (Woods al. 2017)
Headphone test. We used the headphone test developed by Wood et al. (Woods et al. 2017). which used as a standard pre-screener for high-quality auditory psychophysics data-collection procedures (Milne et al. 2021). The test is designed to ensure that the participants are wearing headphones and are able to perceive subtle differences volume. The task consists of a forced choice task, in which three consecutive tones are played, and the participant has to identify which of them is the quietest. Crucially, these tones are constructed to exhibit a phase cancellation effect when not using headphones, and therefore making it difficult for non hea-head Participants had to answer out trials correctly to pass this test. An example trial shown in Figure
to identify the quietest tone.
15

---

Published as a conference paper at ICLR 2023
How similer are the activities following two videos? (2 85)
diffiiult to choose between the options, don't worry, and just give what you intuitively think the right answer.
Play video
Play video
Completely Diesimiar
(1) Dissimiler
(2) Somewinst Dissimlar
Neither Similor Diesimilr
(4) Somewhat Similar
(5) Very Similarr
(6) Completely Similer
Figure 8: Screenshot from the similarity judgment task over video pairs.
B.4 SIMILARITY JUDGMENTS
In the present work, collected similarity judgments across audio and video datasets. Each dataset comprised 4,950 unique pairs corresponding to the number unordered subsets that contain two distinct objects (i.e., excluding self-similaritty, within set of 100 stimuli. We did not collect similarity judgments over the three datasets of images, as these were provided in Peterson et al. (2018) (and used here with permission). The experiments proceeded as follows: upon completion of the consent form and the pre-screening tasks, participants received instructions regarding the main experiment:
Audio. In this experiment we are studying how people perceive emotions. In each round you will be presented with two different recordings and your task will be to simply judge how similar are the emotions of the speakers.
Video. In this experiment we are studying how people perceive activities. In each round you will be presented with two different videos and your task will be simply judge how similar are the activities them.
The instructions then continued as follows:
You will have seven response options, ranging from 0 ('Completely Dissimilar') to 6 ('Completely Similar'). Choose the one you think is most appropriate. Note: no prior expertise is required to complete this task, just choose what you intuitively think the right answer.
The quality of your responses will be automatically monitored, and you will receive bonus at the end of the experiment in proportion to your quality score. The best way to achieve a high score is to concentrate and give each round your best attempt.
The experiment will begin now. You will take up to 85 rounds where you have to answer this question. Remember to pay careful attention order to get the best bonus!
16

---

Published as a conference paper at ICLR 2023
As described in the instructions, in each trial, participants rated the similarity between pair of sounds (how similar are the emotions of the two speakers?) or videos (how similar are the activities scale ranging from 0 (completely dissimilar) to 6 (completely similar) (Figure 8). Overall, participants completed 85 trials on a random subset of the possible pairs. gave them additional performance
in the following two vidcos?)
To further motivate participants to provide good responses, bonus for providing consistent data. Among the 85 trials, 5 trials were repeated for consistency checking. The responses were converted into a performance score by computing the Spearman correlation between the original and repeat ratings. Perfect scores resulted in 10 cent bonus.
B.5 CAPTIONS
We collected free-text captions for the video and audio datasets. Captions for the image datasets were previously collected in Marjieh et al. (2022) and used here with permission. After completing the consent form and pre-screening tests, participants received the following instructions:
Audio. In this experiment we are studying how people describe emotions. You will
presented with different recordings of speakers and your task will be to describe their emotions. In doing so, please keep in mind the following instructions
Describe all the important aspects of the recording.
Video. In this experiment are studying how people describe activities in videos. You will be presented with different videos of activities and your task will be to describe their content. In doing so, please keep in mind the following instructions
Describe all the important activities in the video.
As well as the following guidelines adapted from Marjich al. 2022
Do not start the sentences with "There is" or "There are".
Do not describe unimportant details.
You are not allowed to copy and paste descriptions.
Descriptions should contain at least 5 words.
Descriptions should contain at least 4 unique words.
Note: No prior expertise is required to complete this task, just describe what you intuitively think important as accurately as possible.
The quality of your captions will be monitored automatically and provid- ing low quality and repetitive responses could result in early termination of the experiment and hence a lower bonus.
You will describe up to 50 recordings.
These guidelines were enforced to ensure that participants deliver sufficiently informative captions that are not repetitive. In each trial of the main experiment, participants described a single audio (please describe the emotions of the speaker) or video stimulus (please describe the activity in the video). Overall, participants described up to 50 randomly presented stimuli. To filter out bad participants that tend to deliver repeated responses, in each trial (excluding the first 4 trials) computed the mean edit distance between their current response and all previous responses that they previously provided using the partiar hartial_rat:o string matching. This function returns for pair of input strings a matching score between 0 and 100 (100 being identical strings). Early termination was enforced if the mean response matching score was above 80. The idea here was to prevent participants from copying and pasting the same response over and over again (or varying only slightly).
function in the tuz Python package for fuzzy
B.6 TAGS
For the image, audio, and video datasets, we collected tag data, i.e.. concise labels that describe the salient features a stimulus. To do so, we developed novel tag mining paradigm called STEP-Tag in
17

---

Published as a conference paper at ICLR 2023
Mark the existing tags
picking
apple
Are any tags missing?
Type describing the activity
missing dropdown create entirely Submit
either select your response for a tag by pressing the enter You add more than ane tag.
Flay again
peach
Type more tags
Figure 9: Screenshot of an example tag mining task for videos. The tag "picking" received 5 stars (very relevant), whereas the tag "apple" is flagged (marked as irrelevant).
Dataset (# of stimuli)
mean
std
total
Vegetables (120)
3.2
1.1
385
Furniture (120)
5.2
1.7
627
Animals (120)
8.2
2.7
988
Audio-emotions (1000)
9.1
3.5
9092
Video-activities (1000)
8.5
2.9
8482
Table 2: Mean, standard deviation, and total number of tags collected for each dataset.
which each stimulus was treated as a separate "chain" (see Figure 2 in the paper). When the stimulus was presented for the first time, the participant was asked to provide at least one tag. For the following iterations, we sequenced participants so that each of them had to rate the tags provided by participants from the previous iterations within the same chain. The rating was either choosing between one (not very relevant) to five stars (very relevant), or marking the tag as completely irrelevant by using the flag icon (see Figure 9). Participants could optionally introduce presented to other participants assigned to the same chain. Participants could only provide tags that were already present, and they had be in lower-case letters. To discourage frequent use of long word combinations, a pop-up window appeared if participants used two or more white spaces (i.e., three or more words) to warn that long combinations should only be used when completely necessary. This process continued for at least 10 iterations, after which we checked at each consequent iteration whether the chain was "full". We considered chain be full if latest iteration had at least tags that were rated at least 3 times and had a mean rating of 3 stars. If a chain was not full after 20 iterations, we stopped collecting further iterations. Since each experimental batch lasted for fixed duration of less than
tags that will subsequently be
day, in some cases we did not complete all chains, and a few chains had fewer iterations (3 for vegetables, for animals and for furniture, out of 120 chains each). Our experiment incentivized participants to provide new tags by paying them performance bonus of 0.01 USD for every vo-v (i.e., not flagged) given by other participants. On the contrary, if two or more tags of the same participant were flagged by others, the participant was excluded (the participant received a warning after the first flag). We provide summary statistics on the number of collected tags Table
After accepting the consent form and passing the pre-screening tasks, participants received introduc- tory instructions regarding the main experiment:
Images. Rate Tag animal//uumiurr/eggttabslss Thanks for participating in this game! In this game you will:
18

---

Published as a conference paper at ICLR 2023
Watch images of animail/fumiuure/eeeebbblls Rate tags that other players have given.
Add new tags that you think are missing.
Audio. Rate & Tag emotions! Thanks for participating in this game! In this game you will:
Listen speech fragment and focus on the emotional content of the record- ing.
Rate tags that other players have given.
Add new tags that you think are missing.
Video. Rate & Tag activities! In this game you will:
Watch a video and focus on the activities happening.
Rate tags that other players have given.
Add new tags that you think are missing.
Participants then received further instructions regarding the rules of the game
Images. After watching the aninal/furmiuure/eeettaale you will tags given by other players that describe the aninal/furnitur///ggtaaa You should rate the relevance of each tag by clicking the appropriate amount of stars (I star not very relevant, stars very relevant). If you think that the tag is a mistake or completely irrelevant, you should flag it by clicking the flag icon. If you are the first person seeing this mninal//umiturr/eeetbbbb add your own tag that is relevant to describe the miimal/furniarr//eeetabll Your tag will then be rated by other players who are playing the game simultaneously.
you may see no previous tags. You can also
Audio. After listening to the recording, you will see tags given by other players that describe the emotions in the speech fragment. You should rate the relevance each tag by clicking the appropriate amount of stars (1 star not very relevant, 5 stars very relevant). If you think that the tag is mistake or completely irrelevant, you should flag it by clicking the flag icon. If you are the first person listening to previous tags. You also add your tag that is relevant to describe the emotions the speech fragment. Your tag will then be rated by other players who are playing the game simultaneously.
this speech sample, you may
Video. After watching the video, you will see tags given by other players that describe the activities in the video. You should rate the relevance of each tag by clicking the appropriate amount of stars (1 star not very relevant, 5 stars very relevant). If you think that the tag a mistake or completely irrelevant, you should flag it by clicking the flag icon. If you are the first person watching this video, you may see no previous tags. You can also add your own tag that is relevant describe the activities in the video. Your tag will then be rated by other players who are playing the game simultaneously.
Finally, participants received the following guidelines regarding the tag input and the bonus scheme:
Keep tags short. A word like "green grass" should rather be submitted as "green" and "grass", whereas a compound word such as "red wine" cannot be separated, since "red wine" means something different than just "red" and "wine".
Bonus rules.
If the tag you provide gets rated relevant tag (i.e., not flagged) by other players
If your tag is unique and have not been introduced by others
19

---

Published as a conference paper at ICLR 2023
Modality
STEP
Captions
Audio
230
187
Video
264
291
Table 3: Median of overall participants' time spent per stimulus (in seconds).
Note:
Simply writing many and irrelevant tags not good idea because other players might flag your tag. Your experiment will terminate early if there are too many red flags!
Please try to use a variety of words to describe the animal / furniture / vegetable emotion in the speech fragment / activities in the video, and use the entire star rating scale for your responses.
B.6. VALIDATING STEP-TAG
We conducted a small, exploratory ablation study to validate STEP-Tag as a procedure for collecting diverse, accurate, and informative tags. First, we compared using multiple tags from the last iteration of STEP-Tag to using just a single randomly-selected highly-rated tag from the last iteration. We found that using single tag greatly decreased correlation with human similarity (i.e., for the video dataset, the west-performing method on multiple tags had a correlation of r = 0.74 while the best- performing method single labels had correlation of 0.35). Second, compared tags from the first iteration of STEP- (equivalent to collecting tags without an adaptive procedure) to tags from the last iteration. We found that using first iteration tags greatly decreased correlation with human similarity (i.e., for the video dataset, the 'Tags CNNB mean (no split)' method, the correlation from the last iteration was was r = 0.56 from the last iteration and 0.38 from the first iteration). Finally, we extracted the Kinetics-200 labels for each video compare the tags from STEP -ag against the kinds of labels typically collected for machine learning datasets. We found that using labels decreased the correlation with human similarity (i.e., the best-performing method on pipeline tags had correlation of 0.74 while the best-performing method dataset labels had correlation of 0.64).
0.74 and from the first iteration it was 0.44; for 'Tags overlap'
B.7 DURATION OF STEP-TAG AND CAPTIONS
To compare STEP-tag and captions, we computed the median of overall participants' time spent per stimulus (see Table 3). The times were only collected for the audio and video modality (captions for the image datasets were already collected by Marjieh et al. (2022)). We see that both methods consume roughly similar amounts of time, which is desirable as our analysis suggests that in some domains (e.g., video) tags yield the best results whereas in others (e.g., audio) captions do.
C PREDICTION METHODS
We used main types of methods predict human similarity judgments. The first class ("DNN- based methods", described in section C.I make use of pre-trained embedding models. In the second class of models ("Word Frequency Analysis methods". described in the section C.2) simple feature extraction techniques are used instead of pre-trained deep learning models. Figure depicts schematic overview all prediction methods that we used.
C.I DNN-BASED METHODS
The DNN-based methods use various embeddings and deep learning representations to predict human similarity judgments. These methods could be further split into three groups based the kinds of input data they process, namely if they use a single sensory modality that either image, audio or video ("unimodal models": subsection C.I.I). or text that is either tag captions ('text embeddings"; see subsection C.I.2), or use both ("multimodal models"). In addition, we also tested the performance of "stacked" representations, where the sensory and textual embedding of select
20

---

Published as a conference paper at ICLR 2023
number of models concatenated into single long embedding. Overall, the computation time of embedding methods took about two weeks on an x x1.16xlarge Amazon Web Services instance with 64 vCPUs and 976 GiB of memory.
C.1.1 UNIMODAL DNN-BASED METHODS
Table 4: All 30 image baseline models occurring in the top 50 best models reported in Figure 3
Model name
Average score
SD score
Top 1 accuracy
Number
parameters (M)
Swin ConvNeXT
0.66
0.06
81.52
23.37
0.64
0.07
N/A
348.15
NE-ResNet
0.62
0.04
80.65
23.51
NFNet 10
0.61
0.08
82.75
32.77
ResNetV2
0.60
0.11
N/A
928.34
NF-RegNet
0.59
0.05
79.29
9.26
VGG16
0.58
0.11
73.35
134.27
VGGI9
0.58
0.11
74.21
139.58
ViT
0.58
0.12
75.95
6.16
10
ResMLP
0.57
0.07
83.59
128.37
11
Twins-SVi
0.57
0.06
81.68
23,55
12
Twins-PCPPV
0.57
0.04
81.09
23.59
13
VGGI3
0.57
0.11
71.59
128.96
14
CaiT
0.57
0.04
82.19
17.18
15
VGGI1
0.57
0.10
70.36
128.77
16
gMLP
0.56
0.06
79.64
19.17
17
PIT
0.56
0.03
78.19
10.23
18
DeiT
0.56
0.03
72.17
5.52
19
ConViT
0.56
0.03
73.11
5.52
20
TNT
0.56
0.03
81.52
23.37
21
CoaT
0.55
0.04
78.43
5.35
22
gMixer
0.55
0.05
78.04
24.34
23
XCIT
0.55
0.04
82.57
11.92
24 IG ResNeXt
0.53
0.13
85.44
826.36
25
Visformer
0.52
0.02
82.11
39.45
26
RepVGG
0.52
0.11
80.21
81.26
27
CLIP image
0.50
0.11
N/A
102.01
28
JXNesT
0.50
0.07
81.42
16.67
29
ECAResNet
0.47
0.14
80.45
28.11
30
DenseNet
0.47
0.12
74.74
6.95
Note. Performance accuracy on ImageNet was based on Wightman (2019) for all models.
and was not available
Image models We used 560 pre-trained models from the Pytorch Image Models (mm) repository [Wightman] 2019). We chose this repository as it contains an extensive and highly diverse set of pre-trained models terms of architecture backbones, model sizes, and training sets. The repository includes models published from 2014 4 to 2022 that use various training sets (such ImageNet k. ImageNet21k k, Instagram, etc.), training procedures objectives (e.g., pre-training, fine-tuning, self- supervision, weak supervision, etc.) and architectures (c.g.. VGG, ResNet, Inception, Transformer, etc.). The repository also reports various evaluation metrics for each model (e.g., their ImageNet performance).
For each model, we computed the embedding from the last layer (typically before the final softmax layer; see below and Figure 10 for preliminary analysis for the effect of layer depth in audio models). We then computed the cosine similarity between pairs of embedding vectors to produce a similarity matrix. The entire list of the performance of all models is detailed in the OSF repository
21

---

Published as a conference paper at ICLR 2023
Models
0.5
hubert-base
wav2veclarge_110g
hubert-large
wav2veckarge__gen
0.4
hubert-large, _as
wav2vec-aarge_ _V0k
hubert-xiarge
wav2veclarge__.e00K1000
0.3
Score
hubert-xtarge_et
waw22eeclarge_.N00K.10m
0.2
wav2vec-base
www2veeckaree_w60k.Ko0
wav2vec-base 100h
waw2veccaarg__XLSRS
0.1
wav2vec-base_ 10m
wavim-base
wav2vec-base 960h
wavim-base+
0.0
Close output
Close input
wav2veclarge_100h
wavlm-large
Layer depth
Figure 10: Scores for individual layers of audio models scaled to the total number of layers. Models colored by their meta architecture.
associated with this paper | Table presents additional details for the top 42 image baseline models in Figure 3 including their average score (correlation to human judgments) across the three image datasets, the standard deviation (SD) of this score (across datasets, repeated runs and available model parameters Wightman (2019)) their ImageNet accuracy, and their number of trainable parameters.
Figure A shows the correlation to human similarity as function of the number of parameters for all 569 models. In general, we found that models that have more parameters perform better (Figure 4A). Plotting all the embedding technique correlations against the number of training parameters of 0.39. p 0.001). However, one possible explanation for this could be the improved performance of newer models, which typically have more parameters, on various computer vision tasks. To test this, we computed the performance (i.e., correlation with human similarity) of the various models as a function of their accuracy on ImageNet (Deng et al.) 2009) - which was provided in [Wightman| (2019) for all models except for CLIP (whose implementation came from a different repository)
their respective models showed statistically significant positive correlation
summarized in Figure We found a positive correlation between the two metrics (r = 0.26, p < 0.001), though with some clear exceptions. For example, the vision transformer BETT [ao et al. 2021 and the convolutional architecture EfficientNet (Tan & Le 2019) achieved high accuracy on ImageNet but performed poorly on human data. On the other hand, the vision transformer Swin (Liu al. 2021 and the convolutional architecture ConvNext (Liu et al.) 2022) both performed well on ImageNet and human similarity. This suggests that architecture and number of parameters are better predictors of similarity judgments than performance on ImageNet. Further analysis is required to determine what kind of architectural components actually contribute to more human-like performance (Langlois) et al. 202T).
We used all pre-traines wav2vee 2.0 Baevski et al., 2020) and HuBERT (Hsu et al. 2021) models available in torchaudic o YYang et al. (021) We also extracted embeddings from WavL.M (Chen et al.) 2021) and data2vec audio models (Baevski et al. 2022). Furthermore, we used additional wav2vee 2.0 and HuBERT models that were either special ized on emotion recognition
Audio models
or speaker identification (wen Yang et al. 2021 Wagner et al., 2022: Ravanelli et al. 2021 The
performance HuBERT, wav2vec 2.0, and WavLM models is shown in Figure 3B, Additional details
about the models are displayed in Table
In addition, we explored the correlation between the audio models and human similarity data as function of the layer in the model. Earlier literature has suggested that similarity to human
representations may depend on the layer of the model (Kell et al. 2018 Yamins et al. 2014 Yamins
22

---

Published as a conference paper at ICLR 2023
Table 5: All audio baseline models used in the analysis.
Model name
Emotion correlation
Number of parameters (M)
wav2vec 2.0 1v60k (100h) wav2vec 2.0 1v60k (960h)
0.49
317
0.49
317
wav2vec 2.0 Iv60k
0,51
317
wav2vec 2.0 1v60k (10m)
0.51
317
5
HuBERT xlarge ASR
0,45
1000
HuBERT xlarge
0,46
1000
HuBERT large ASR
0.46
300
wav2vec 2.0 large XI.SR33
0.47
317
9
HuBERT large
0,46
300
10
wav2vec 2.0 (Audcering, emotion)
0.49
317
11
HuBERT base
0.41
90
12
WavLM large
0.46
316.62
13
HuBERT base (superb, emotion)
0.42
90
14
HuBERT base (superb, speaker)
0.42
90
15
WavLM base+
0.41
94.70
16
wav2vec 2.0 base (960h)
0.38
95
17
WavLM base
0.39
94.70
18
wav2vec 2.0 base
0.34
95
19
wav2vee 2.0 base (10m)
0.34
20
wav2vec 2.0 base (superb, emotion)
0.34
95
21
wav2vec 2.0 base (superb, speaker)
0.34
22
wav2vec 2.0 base (100h)
0.32
95
23
HuBERT large (superb, emotion)
0.29
300
24
HuBERT large (superb, speaker)
0.29
300
25
wav2vec 2.0 large (100h)
0.32
317
26
wav2vec 2.0 large (superb, emotion)
0.31
317
27
wav2vec 2.0 large (superb, speaker)
0.31
317
28
wav2vec 2.0 large (960h)
0.31
317
29
wav2vec 2.0 large (10m)
0.31
317
30
data2vec audio large (960h)
0.31
313.28
31
data2vec audio base (100h)
0.23
313.28
32
data2vec audio large (100h)
0.23
313.28
33
data2vec audio large (10m)
0.21
313.28
34
wav2vec 2.0 (SpeechBrain, emotion)
0.11
95
35
data2vec audio base (960h)
0.16
93.16
36
data2vec audio base (10m)
0.15
93.16
2020). We expected that the layers closer to the input of the model (where the representation is more low-level) be less predictive. In general, we found that this was the case (Figure [O].. In some variants of wav2vec, however, intermediate representations performed better, possibly due to the misalignment of the training task of wav2vec with the emotion task. This analysis confirms the choice we made in the paper to mostly use the last two layers of the models. Preliminary analysis of the image and video models also explored different layers, but the results were similar to those we presented in audio, and are therefore not reported here.
Video models We extracted embeddings from the 'Slow' (a 3D ResNet; see Feichtenhofer et al. (2019)) Slowfast (a 2-path model with one path capturing semantics and the other capturing fine details; see Feichtenhofer et al. (2019), and X3d (a model that initially start as simple 2D image classifier but is expanded in several axes; see Feichtenhofer (2020)) architectures implemented in pytorchvide (Fan et al., 2021). All video models were pre-treined on the Kinetics-Ae dataset (Kay et al., 2017). The performance of the models is displayed in Figure 3C. Numeric correlation values are detailed in Table along with model accuracy (Top1 and Top5) on Kineticy-400, and the number of parameters in each model. The accuracies and parameter counts are listed as reported in
23

---

Published as a conference paper at ICLR 2023
Fan al. 2021). As with previous modalities, the number of parameters appears to be positively correlated with correlation to human similarity.
Table 6: All video baseline models used in the analysis.
Model name
Correlation
Kinetics-400
Kinetics-400 Top5 Acc
Number of parameters (M)
Top1 Ace
1
Slowfast r50
0.65
76.94
92.69
34.57
2
Slowfast r101
0.64
77.90
93.27
62.83
3
Slow 150
0.61
74.58
91.63
32.45
4
X3d M
0.53
75.94
92.72
3.79
5 X3d S
0.49
73.33
91.27
3.79
6
X3d XS
0.48
69.12
88.63
3.79
C.1.2 TEXT EMBEDDING METHODS
Caption text embedding, Since there are multiple captions per stimulus, an aggregation procedure had to be applied to produce single embedding vector for each stimulus. In our main analysis, for each stimulus, we extracted the embedding for each associated caption and averaged these embeddings together before computing cosine similarity between the mean embeddings. We also tried an alternative approach of concatenating the captions together into single paragraph, which we then passed through the LLMs compute single embedding per stimulus. We found that this did not consistently improve performance and in many cases even decreased it, though we note that we did not experiment with different permutations of the concatenated captions, nor did we extensively study other ways to combine them together. Future work could explore other techniques for pre-procss captions and aggregating representations from multiple captions in ways that would improve correlation with human similarity judgments.
Tag text embedding. We experimented with several algorithms for computing similarity between sets (or multi-sets) of tags. The algorithms described in this section all involve using ConceptNet NumberBatch (CNNB) (Speer et al. 2017) as the embedding backbone for turning discrete tags into continuous vector representations. For each stimulus, took the tags remaining in the final iteration, and tested whether they were found in the dictionary for our embedding model. If tag was not found and if it contained spaces, we tried correct the spelling before trying to look up in the dictionary again. If a tag contained spaces, we split into individual words, correct their spelling, and averaged together the embedded representations of those words that were found in the dictionary. Tags that were not found even after spelling correction and splitting were excluded from the set and did not contribute to the final representation. For the methods marked '(no split)' we did not split multi-wond moldi tags, instead we just excluded multi-word tags that were not found in the embedding model dictionary. In the following, we describe the different techniques used to generate predictions based on tag embeddings.
Tags CNNB overlap. For each pair of stimuli, we counted the number of 'almost identical' tag embeddings, defined as every respective element of the two embeddings being less than a certain threshold apart (in our case, this threshold was 0.1). We then set similarity for that pair of stimuli to be this count, i.e., the number of 'almost identical' tags, normalized by the total number of tags across the respective two sets.
Tags CNNB quantized. This method involves quantizing tags using cosine similarity to find the number of unique tags. For each pair of stimuli, we counted the number of tags assigned to the first stimulus that had cosine similarity greater than certain threshold (in our case, this threshold was 0.7) to at least one tag of the second stimulus (call this value NA) and vice-versa (NB). The minimum of these two values is the number of unique, shared tags between the two sets (min (N AN NB)). The total number of unique tags across the two sets is then the total number of tags in each set (T'A + TTB) minus the maximum number of shared tags (max (N A, NB)). We compute similarity as the ratio of the min Na.
number of unique shared tags to the total number of unique tags, SAB = TA+IB - (NA.N)) For example, suppose the two sets of tags are A fa, b, c, 9} and B {a, b, d, e), so TA = TB 4, and
that a, e have cosine similarity of 0.8. The number of tags from set A found in set B is NA = 3, and
those from B found in A is NB 2. The number of unique, shared tags min(NA NB) = - (since
24

---

Published as a conference paper at ICLR 2023
{a. b, et can be represented by {a, b)), and the total number of unique tags is 4 4-43 5 (since {a, b, c, 9, a, b, d, e} can be represented by {a, b, d, e, g. ) The assigned similarity is then SAB =
Tags CNNB mean. The set of tag embeddings for each stimulus were averaged together form single embedding assigned to the respective stimulus. We then computed cosine similarity on the embeddings of each pair of stimuli.
Tags CNNB mean (no split). Same as above, but without splitting multi-word tags (i.e., ones that contain spaces) during the embedding process.
All spelling corrections in the algorithms listed above were performed using the Python package Pyspel 1 checker taking the top corrected recommendation returned by the spell checker in each case.
Tags caption Roberta (SimCSE). Additionally, for the images datasets, we experimented with converting sets of tags into captions and then using those captions with our best-performing LLM (' cop-simccse-rbbett-llee
the same way do with user-generated captions. To convert set of tags into a caption, we joined the set of tags with commas and prepended them with the phrase "This an image of"".
C.1.3 OTHER DNN-BASED METHODS
For the image datasets, we also considered several other methods that made use DNNs but do not fit into the categories described above.
GPT3 prompting We experimented with prompting GPT3 (Brown et al. 2020), a large pre-trained language model, directly output similarity judgments as a lexl-completion problem rather than having to access model embeddings as we did above. We used a few-shot prompting approach where in each prompt we included three context examples of pairs of tag sets and their associated similarity rating. We then provided the pair of tag sets for the two images that we wanted to get similarity rating for but left the rating empty for the model to
Here is an example prompt with the GPT3 response bolded and in square brackets:
People described pairs of images using words. How similar are the two images in each pair on a scale of 0-1 where 0 completely dissimilar and 1 is completely similar?
Here the descriptions of image one: tortoise, slow, protected. shell, turtle, scaly, old, cold-blooded Here are the descriptions of image two: monkey, ape, mammal, black and white, hairy, agile, primate, smart, tree-dwelling Rating: 0.05
Here are the descriptions of image one: rhinoceros, horn, gray, standing, heavy body, endangered, wild, africa, african
Here the descriptions of image two: tiger, open mouth, stripes, feline, predator
Rating: 0.27
Here are the descriptions of image one: goat, eye, leg
Here the descriptions of image two: mammal, wide-nosed, mandrill, primate, baboon, smart Rating: 0.19
Here are the descriptions of image one: black, primate, mammal, hairy, chimpanzee, africa, african, great ape, smart, omnivore Here are the descriptions of image two: zebra, striped, two-toned, wild, staring, mammal, equine, herd animal, africa
Rating: [0.14]
We repeated this four times for each pair of images in each image dataset with different set of context examples during each repetition and averaged together the GPT responses to get a final
https::/pyspell checker. readthedccs. hheeeeee...a..taedecses. http://////ee/e/e//eaaa/taaee
25

---

Published as a conference paper at ICLR 2023
similarity prediction for each pair. In total, creating the context examples required having access to human similarity judgments over only 12 pairs images. We found that this approach yielded surprisingly good predictions, with average correlation of r 0.62 across the image datasets. We believe this approach merits future investigation to determine whether prompt engineering can further increase the performance.
Image captioning models We experimented with using pre-trained image captioning models to generate captions for our images and then using those captions with our best-performing LLM ('sup- simese-roherta-large the same way we do with user-generated captions. We used three pre-trained image captioning models from HuggingFace ('lamiigo-miin', 'vil-b32-finetuned--e gpt2-image-cape iinge-captiooingg) ) to generate text descriptions for our images. However, the performance was quite poor with average of r 0.29 across the three models. As a result, ( language-bast based methods cannot easily be reduced to O(1) even when Noomain-relevane pre- pre-trained caption models are available.
and "vit-
C.2 WORD FREQUENCY ANALYSIS METHODS
In this work, we also conducted an additional evaluation of prediction models beyond embedding- based techniques (described in the previous section). Specifically, we compared the predictions of embedding- based models, which utilize deep learning representations, with those of traditional methods of text mining.
Before the word frequency analysis, we performed the following initial pre-processing steps
For caption data, we concatenated all the captions describing the same stimulus into a single long "document."
For tag data, we wanted to prioritize tags that appeared earlier in the lag-iniin chains and were rated higher. To that end, we gathered all tags from all iterations and duplicated tags from a given iteration based on the ratings they received. For example, if the tag "tomato" received three stars, then would add the repeated tokens "tomato, tomato, tomato" to the aggregated list ("document"). In a given iteration, flagged tags are removed, but if they are rated later, then they are included. The total number of repetitions per token is equal to the sum of all the stars they received in all iterations. As result, each token is repeated multiple times, which we take into consideration in consequent analysis.
For the next steps, we used the Matlab text analytics toolbox F Unless otherwise specified, we used default parameters for all functions. To generate similarity matrices, we applied the following methods:
Co-oceurrence method. In this approach, we simply counted the number of repeated pairs of words in documents and j and normalized by the total number of pairs. Formally, we use wi to denote the word list of a document i. Let wi.s be the k-th word in the w; list of words, and let |wy] denote the length of the list. We denote by 8(c, d) the indicator function that returns if and only if the word e identical to the word d, and 0 otherwise. We computed the co-occurrence score S(w,, wy) according to the following formula:
Sww, wy)) Ee 5, w5y1)
[wa [zy
We suggest using this method only with tags and not with captions.
Co-occurrence-rep. This method was applied only to tags. We used an identical procedure to the Co-occurrence method, except that we did not separate the words within a tag as separate tokens and instead treated the entire tag (that may include multiple words)
single token.
Rouge score. this approach, similarity was estimated by computing the rouge score of the word lists associated with each pair of documents. The Rouge score was computed using rougeEvaluationsca. (Rouge 2004). We suggest using this method only with tags and not with captions.
html
26

---

Published as a conference paper at ICLR 2023
The following methods make use of tokenized data and pre-processing procedure that we found effective. Pre-processing was applied to both tag and caption data and tokenization was performed as follows:
We separate all text into single words by applying the tokeni tokemiiedDoccument function.
We added part of speech information using the addP adddeaattt@SSeeeheeeteaee function.
We performed Lemmatization using the
function.
We erased punctuation from the token using the erasePunctuatioon function.
We removed stopwords using the SemoveSttoooods function.
We removed words with less than characters or more than characters.
We created bag of words representation of each tokenized document using the bagOfWords function.
We also removed words that were not present in more than two documents using the InfrequentWords function.
With the results of these pre-processing steps, then computed similarity matrices based the following methods:
bm25S. We used bm255 compute similarity between documents (Barrios al. 2016) using Matlab's bm2 5 Simiiiiiiiiiii larity function. This function represents TF-IDF-like retrieval functions used in document retrieval. We used a variant that has normalization function that properly handles documents with long list of words.
if-idf-cosine. We computed pairwise cosine similarities between document pairs using the TF-IDF matrix derived from their word counts and Matlab's cos cosines imi larity function.
C.3 SUPERVISED METHODS
Several previous studies investigated improving correlations by applying and fine-tuning simple linear transformations to embedding vectors 2 Wz where W diag(W1..... w.) via a cross- validated ridge regression procedure that could be fit to ground-truth similarity judgments. The parameters of the diagonal reweighting matrix W fitted training subset of stimuli and used prediet similarity pairs in a held-out validation set [Peterson et al. (2018); Marjieh et al.) (2022). To be consistent and make results comparable, here we report the results of performing this 6-fold cross-validated linear transformation (LT-CCV) on the model embeddings and datasets considered in this work. The analysis was carried out using the Rid RideeV package from the scikit-leal Python library |Pedregosa et al.) (2011). Results with both normalized ('LT CCV (norm)') and un- normalized ('LT CCV') regressors are shown in Figure IT, see Ri dgeCV documentation for details on normalization 10 We see that the linear transformation does not consistently improve performance stacked embeddings, but does frequently improve performance when applied to caption embeddings. Due to their instability and risk of overfitting, we do not use these methods in our main analysis.
(and can even decrease it) when applied to many of the modality-hand
http:p///ookk...anoo.ss/saallaas near neaar_ode!
Ri RidgeCV. html
27

---

Published as a conference paper at ICLR 2023
image
audio
video
0.8
0.6
Raw
0.4
0.2
0.8
0.6
LT CCV
Score
0.8
CCV (norm)
0.6
0.2
Caption-
Stacked
Baseline
Caption- only
Stacked
Baseline
Caption-
Stacked
Baseline
wav2vec 2.0 large lv60k (10m)
Slow r50
ConwNeXT base in22k
wav2vec 2.0 large lv80k (100h)
Slowfast r50
Swin base patch4 window7 224 in22k
Baseline model
wav2vec 2.0 large lv60k (960h)
Slowfast r101
Swin large patch4 window7 224
wav2vec 2.0 large lv60k
X3d M
VIT base r50 s16 384
wav2vec 2.0 large XLSR53
huge patch14 in21k
Figure II: Effect of fine-tuning model embeddings using small subset of similarity judgments.
28


Reviews:
[{"review_id": "yUP72x_OyqI", "reviewer": "Reviewer_ZARH", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To approximate human judgements without increasing cost and losing reliability, the authors proposed propose a new class of similarity approximation method based on language, namely captions and stacked words. Their method achieved good approximation with only O(N) computation cost. They also developed the novel overall flow to compute representational similarity. They also claim they are going to make the human judgement dataset publicly available.", "summary_of_the_review": "I think this paper should be accepted for the “Human like” evaluation point of view as DNN becomes more sophisticated. I believe their finding is also beneficial for the community.", "strength_and_weaknesses": "[Strength]\nThe concept of the paper is nicely presented for the next AI abundant society where demand for human judgments is more increased. They proposed the novel similarity approximation method with a smaller number of judgements.  \nThey provide the evaluation data in the supplemental material, which is beneficial.\nOverall, I believe what the authors done is quite systematic and very beneficial for the future refinement of the DNN model.\n\n[Weakness]\n\n-\tRepresentational similarity\nFor beginners, the word of representation similarity is not easy to follow. At least, authors should define the terminology in the introduction.\n\n\n-\tThe title\nIn particular, the linkage between title and abstract seems not clear at the first glance.\nPlease consider to update the title and abstract so that readers can easily understand the topic theme.\n\n\n-\tIs English best?\nThere are many languages. In my opinion, I view that authors selected English language to represent the basis of all things in nature. Why English(e.g., language system such as polysemous, grammar, dataset availability…)? Justification of English employment would make the paper better. \n\nRelated to above topic, the paper report “N = 1,492 US participants for the new behavioral experiments reported in this paper.” Are they all Americans and share the same contexts? (e.g., cultures…) \n\n-\tCrowd scouring\nHow long does it take to STEP-Tag? It is good that measure actual time that participants consumed. Comparing with that with the time used for writing captions may strengthen your claim. \n\n-\tDiscussion\nI feel strange because the paper end with “Discussion”. Please consider adding conclusion section or rename the last section as “Discussion and Conclusion”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "YNKRxEZYKke", "reviewer": "Reviewer_hUBS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new class of similarity approximation methods based on language. To collect the language data required by these new methods, the authors also developed and validated a new adaptive tag collection pipeline, which is significantly cheaper than the classical methods. Finally, the authors also develop ‘stacked’ methods that combine language embeddings with DNN embeddings, and find that these consistently provide the best approximations for human similarity across all the three modalities. ", "summary_of_the_review": "The main issue of the paper is the novelty of the proposed method. Based on technical novelty and insufficient experiments, I don’t think the current version meets the standard of the ICLR. ", "strength_and_weaknesses": "Strength:\n1. The paper is overall well-written and easy to follow.\n2. Multiple datasets are evaluated for the proposed method.\n3. Multiple modalities are explored and evaluated on the proposed method.\n\nWeakness:\n1. The technical contribution of the proposed method is weak.\n2. No baselines are compared to the proposed method in the experiment section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FLYixTi1fr3", "reviewer": "Reviewer_AJDz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider the task of approximating human didactic\nsimilarity judgments over N pairs of (images/audio/text). While high\nquality human judgments over all pairs is ideal, (N choose 2) is too\nbig for large N. The authors consider gathering N descriptions of the\nobjects (tags or captions), extracting text representations of those\ndescriptions, and then computing text-text similarity. Experiments\nacross a large number of models and datasets reveal the surprising\neffectiveness of computing text-text similarities.", "summary_of_the_review": "Overall, this work address an interesting task/approach ---\napproximating pairwise similarity judgments over N objects with only\nO(N) annotation via language models. While this setup has been\nconsidered by Marjieh, this work presents more experiments that show\nthe text representations are surprisingly strong. They consider a very\nlarge number of models, modalities, etc. The released tag sets would\nlikely be useful to someone. But, a handful of missing additional\nexperiments, which are arguably in-scope for the present work, are not\npresented. And, prior work has already made the observation that LLMs\ncan approximate similarity judgments across non-textual modalities,\nwhich appears to somewhat limit the novelty of this work.\n\n\nAfter response:\n\nThe authors seemingly ran all of the experiments I suggested and have incorporated them into the paper. Amazing :-)\nFurthermore, the authors clarified the difference with prior work.\nAll my concerns are addressed, so I raised my score from 5-->8", "strength_and_weaknesses": "Reconstructing pairwise human similarity judgments across N objects is\nan interesting task, and the author's idea of gathering textual\ndescriptions is clever. The performance of the stacked models (which\ndepend on domain-specific pretrained models and text-text\nsimilarities) and text-only models are surprisingly strong across a\nvariety of domains --- this finding has implications for the relative\nsurprising representational strength of text. The annotation process\nthey describe for their tag corpus is interesting --- multiple rounds\nof annotations are undertaken with crowdworkers fixing each-other's\nerrors. The authors consider a large number of models across many\ndomains --- the benchmark experiments are extensive, and the \"size\nvs. accuracy\" plots in the end are cool. Figure 5 is amazing --- few\nworks offer such clear practical guidance.  Finally, I liked the\nhopeful message in the end --- that a combined multimodal approach\nseems to work best, so don't completely discard the domain-specific\nrepresentations.\n\nIt should be noted that the cited work where some of the data is from,\nMarjieh et al. 2022, is quite similar in the sense that it also\nproposes to estimate human similarity judgments. This work appears to\nextend that work by exploring more types of models and their\ncombinations.\n\nMy main concerns are that this set of experiments suggests some clear\nnext setups that I don't nesc. think are out of scope for this work.\nSpecifically:\n\n- If the goal in practice is to reconstruct human judgments --- I\n  would have liked the authors to compare against a supervised setup.\n  How much of the remaining misalignments can be fixed by gathering n\n  << N^2 additional pairwise judgments and then training a supervised\n  model ontop of the models?\n\n- I would have liked to have seen more experiments with even less\n  supervision --- is it possible to gather n < N captions, and train a\n  model on those very few pairs to map to text, and then use the\n  resulting modality --> text model as input to the models shown here?\n\n- I would have appreciated more discussion of tags vs. captions for\n  the images. In figure 3A, I assume that the text only methods use\n  captions unless specified otherwise. But --- because the tags were\n  not fed to, e.g., BERT/RoBERTa, I can't tell if the performance\n  gains are due to the full captions being used, or if the gains are\n  due to the more expressive models. A trivial linearization of, e.g.,\n  \"A photo of a tag1, tag2, and tag3\" handed to a LLM would have been\n  a nice comparison to see.\n\n- Finally, I would have been interested in a fully LLM setup where\n  instead of pairwise cosine similarities, both text pairs are fed to\n  a very large LLM (e.g., OPT or GPT3) and the model outputs a likert\n  rating as text.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BnIPXCoBp_s", "reviewer": "Reviewer_kd7y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors make the observation that representations learned by DL models produce proximity scores very different from human evaluations. They introduce a simple and scalable technique to make the human and model produced similarity scores closer. Essentially, text descriptions or tags for various input data points (across modalities) are passed through LLMs (or word-frequency methods) and the resulting representations used for proximity scores. Stacked representations combining the existing model and the text description representation is shown to consistently match human similarity metrics better.", "summary_of_the_review": "- Well motivated problem \n- Clearly described technique that is scalable, easy to implement\n- Techniques like these that are easy to implement and help with interpretability are of great use in the small-data regimen (where the bulk of us are). I would like to see this paper at ICLR.", "strength_and_weaknesses": "Strengths: I think this is a well-motivated problem. Learned representations are often used as features in the small-data regimen or sometimes directly for getting proximity scores in an AI setting. This paper address the human interpretability of these representations by (i) confirming that human similarities and proximity scores from models can vary a lot, (ii) text-descriptions or tags can be leveraged and stacking these representations with the model-learned representations can help. I also appreciate that the technique is scalable and in many cases not that much of an overhead to implement. I appreciate the arguments in the related text that leverage cognitive science literature. In addition, the paper is easy to follow.\n\nWeaknesses: The paper doesn't have too many weaknesses. I was wondering if we could get some numbers on if the stacked representations help in additional downstream tasks like say classification (i.e. does the performance on imagenet improve if you use imagenet + text). However, I understand that this can be significant undertaking and do not want to base my review on this experiment but it is a potential future direction.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster