DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding
Authors: Xiao-Yu Guo
Yuan-Fang Li
Gholamreza Haffari
Venue: EMNLP 2023 Main (2023)
Keywords: Question Answering; Social Intelligence; Multimodal Learning

Abstract: Social intelligence is essential for understanding and reasoning about human expressions, intents and interactions. 
One representative benchmark for its study is Social Intelligence Queries (Social-IQ), a dataset of multiple-choice questions on videos of complex social interactions. 
We define a comprehensive methodology to study the soundness of Social-IQ, as the soundness of such benchmark datasets is crucial to the investigation of the underlying research problem. 
We define a comprehensive methodology to study the soundness of Social-IQ, as the soundness of such benchmark datasets is crucial to the investigation of the underlying research problem. 
Our analysis reveals that Social-IQ contains substantial biases, which can be exploited by a moderately strong language model to learn spurious correlations to achieve perfect performance without being given the context or even the question. 
We introduce DeSIQ, a new challenging dataset, constructed by applying simple perturbations to Social-IQ. 
Our empirical analysis shows De-SIQ significantly reduces the biases in the original Social-IQ dataset. 
Furthermore, we examine and shed light on the effect of model size, model style, learning settings, commonsense knowledge, and multi-modality on the new benchmark performance. 
Our new dataset, observations and findings open up important research questions for the study of social intelligence.

DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding
Xiao- Yu Guo, Yuan-Fang Li, and Gholamreza Haffari
Faculty of Information Technology, Monash University, Melbourne, Australia mi..g.....ggoomeeereeeoe gholamreza. yemffasi)mmonsh.... edu
{xiaoyu.gus guo, yuanfang.
anana li,
Abstract
current Al systems. It is multiple choice question answering dataset with multi-modal inputs, includ- ing questions, answer options, vidcos, ctc; see an example in Figure 1. Although Social-IQ contains rigorously human-annotated data, surprisingly, we find even small models like T5-small (Raffel et al., 2020) could easily achieve 100% answer option accuracy (Table 3).
Social intelligence is essential for understand- ing and reasoning about human expressions, intents and interactions. One representative benchmark for its study is Social Intelligence Queries (Social-IQ), dataset of multiple- choice questions on videos of complex so- cial interactions. We define comprehen- sive methodology to study the soundness of Social-IQ, soundness of such benchmark datasets is crucial to the investigation of the underlying research problem. Our analysis re- veals that Social- contains substantial biases, which can be exploited by moderately strong language model to learn spurious correlations achieve perfect performance without being given the context or even the question. We introduce DeSIQ, a new challenging dataset, constructed by applying simple perturbations to Social-IQ. Our empirical analysis shows De- SIQ significantly reduces the biases in the orig- inal Social-IQ dataset. Furthermore, we amine and shed light on the effect of model size, model style, learning settings, common- sense knowledge, and mudti-modality on the new benchmark performance. Our new dataset, observations and findings open up important research questions for the study of social intel- ligence.
The perfect performance of such an underpow- ered model prompted us to conduct further investi- gation to identify its source. Through employing different models and perturbation methods on the answer options, we identify significant biases in the Social-IQ dataset, in which the representations of correct and incorrect options are easily separable. regardless of the questions (Figure 3). Thus, the models are able exploit such shortcut (Jia and Liang, 2017; Jiang and Bansal, 2019) to answer questions with a high accuracy, without necessarily understanding social intelligence.
To debias the Social-IQ dataset, we propose a simple yet effective debiasing approach and present unbiased benchmark DeSIQ, by substituting all the incorrect answer options with correct answer options from randomly selected other questions.
We establish performance baseline on DeSIQ with T5-small and Delphi (Jiang et al., 2021), a language model pretrained with commonsense and social norms knowledge. Given answer options only or question-answers, both T5-small and Del- phi obtain close to random accuracy. By making use of multi-modal inputs, both T5-small and Del- phi achieve an accuracy of up 77%. These results demonstrate that DeSIQ is unbiased and challeng- ing. Interestingly. both models also outperform GPT-3 and ChatGPT, further indicating the chal- lenging nature of the social intelligence understand- ing problem.
1 Introduction
Social intelligence is a long-standing research area in social science and psychology (Thorndike, 1920; Andreou, 2006; Goleman, 2007). It is the capacity to understand and navigate complex social situa- tions. Social intelligence is more than the percep- tion of objects and human actions, as it requires a deeper understanding of human intents and interac- tions behind these actions words.
The study of social intelligence is an emerging area in both the NLP and computer vision com- munities. One representative work, Social-IQ (So- cial Intelligence Queries) (Zadch et al., 2019), is a benchmark dataset measuring social intelligence of
Our contributions
We propose six formally defined methods to iden- tify the bias in Social-IQ. From the answer pertur-

---

Why are they people sitting next cach other?
Their conversation easier that way.
i- i-They-are-olle
Chappy (acc)
The general theme is surprise
(distain face)
Urustrated arguing (sse)
00:04900:114
00: 00:16->000:22
00:31 59
Likee Kylie Jennen
Ohi could get Chaps He womts to ge intu the
-lka, me What about monls fishion, okay?? Because the people that are attracted the van don't really have fasitio seuse and there could be episodes Look, you
Yesh.
show business,
Sheld pet soms views, lnii?
He basalt broks out of jail yet.
"heen did caught Hlomo Oll.
photo with Penn?
you've guita what
morketss telling you
Instagram photo got like two
Whoa.
Q1: How does the man in the middle fool about the
q3: How does the woman react the man in the middle's suggestion?
q4: How does the man en the right react to the woman's suggestion?
man right's idea Kylie Jenner?
a: He feels enlightened hecause was a great idea that did think
She reacts with disdain bocause it is not plausible idea.
He is frustrated bocause he thinks it really bad idea.
i-He-isrmngry becwene tre teeewworeeee womi-ggingg we
She happy hecause is iden.
i: He likes the idea lot and happy she supgosted
al: She extremely surprised because she sees
al: the girl reacted in very excited way
celebrity that she likes very much
at: the the orange
she exited
Figure I: One example in the Social-IQ and DeSIQ benchmark. For Social-IQ, q. a, stand for question, correct and incorrect answer respectively, while a' with yellow background color is the unbiased incorrect answer we substitute in DeSIQ. Different colors represent different persons, including the facial expressions and oral speaking words. The transcripts are three black squares related certain video clips the above.
bation experiments, we find that the bias mainly exists in the answer options.
Number
Training
Development
Total
Video
888
127
1,015
Question
5,328
762
6,090
Correct
21,312
3,048
24,360
We propose DeSIQ, an unbiased, and more chal- lenging multi-modal question answering bench- mark, designed to better measure social intelli- gence for machine learning models.
Incorrect
15,984
2,286
18,270
Table I: Statistics of the Social-IQ dataset. On average, each video has 6 questions; for each question, there are 4 correct answer options and incorrect answer options.
We propose two effective models that outperform the baseline and GPT-3/ChatGPT on our new benchmark. We also make detailed analysis and comparison the performance of these models.
Numher
Training
Development
Total
Video 987
145
1,132
Question
6.159
943
7.102
Correct
6,159
943
7,102
Incorrect
18,477
2,829
21,306
2 Identifying Biases in Social-IQ
Table 2: Statistics the Social-IQ-2.0 dataset. For each question, there is only 1 correct answer option.
2.1 The Social Intelligence Datasets
Social-IQ (Zadeh et al., 2019) is an unconstrained multi-modal, multiple-choice question answering (MCQA) dataset designed to evaluate the social intelligence of machine learning models. It con- tains videos about social interactions, questions and multiple-choice answer options, in which the questions and answer options were crowdsourced. For each video, the context for all questions and answer options includes not only the original video, but also the corresponding extracted audio and au-
tomatically generated transcripts' Detailed dataset statistics are shown in Table 1.
Social-IQ provides two configurations: A2 (2- way, i.e. one correct answer option and one in- correct option for each question) and A4 (2-way, i.e. one correct option and incorrect options for each question) for training and evaluation, in which
'We don't have access to the raw transcript, video and audio data so we use extracted features downloaded from https:// tttpb/ggtttt... wwwmmatsuoleeeeecoooooooooooocce htt.///atstttttttooama Laup//U-///// timodalSDK.

---

model performance is measured using binary and 4-way accuracy respectively.
Most recently, Social-IQ-2.0 was released on- line? with the A4 configuration. Though nearly half of the videos overlap with Social-IQ, almost all questions and answers were newly annotated. Moreover, raw video and audio files have been provided instead of only features in the original Social-IQ dataset. The detailed statistics are shown in Table 2. For simplicity, and v2 represent Social-IQ and Social-IQ-2.0 respectively, which are used interchangeably.
LSTM
LSTM,
LSTM
LSTM,
Fg
MLP
MLP
Sy
Overall architecture of the LSTM base- line (Zadeh et al., 2019). q.a, i,t, denote question, correct answer, incorrect answer, transcript and video features. ry. Ta. ri, ft, Fy are corresponding features ex- tracted using different LSTMs. Dashed squares repre- sent optional input features. Two multi-layer perception (MLP) are parameter-shared. The output will be two scores $1, 82 respectively of the correct and incorrect answer options.
Figure 2:
2.2 Methodology
this section, we propose several experimental settings to identify biases in MCQA dataset. Let q and o' denote two different questions, a and i denote the correct and an incorrect answer option of q respectively, and a' and i' denote the correct and incorrect answer option of q' respectively. We define six methods to identify biases:
the original dataset. With RAWI RAWA, the model should perform significantly worse.
No context and question (NCAQ): the contexts and questions for all answer options are removed. Le., the model is only given all answer options.
2.3 Biases in Social-IQ
We evaluate the A2 (binary choice) configuration and A4 (multiple choice) configurations of Social- IQ, and A4 configuration of Social-IQ-2.0 the experimental settings discussed above, and surpris- ingly, we observe that they are both biased. Below, we describe our detailed analysis and show that Social-IQ contains substantial biases that can be exploited by moderately strong language models. Table 3 summarises the experimental results. In the fully supervised setting, we evaluate the perfor- mance of the LSTM-based model in the original Social-1Q paper (Zadeh et al., 2019) (Figure 2) as well as the more capable T5-small (Raffel et al., 2020), which we use as the encoder to replace the LSTM in Figure 2.
An MCQA dataset should be sufficiently chal- lenging that no model predict correct an- swer when neither the input context the ques- tion not provided.
More Powerful Model (MPM): the model is sub- stituted by a larger, more capable model.
It is plausible induce a performance increase on the dataset when stronger model (e.g. with more trainable parameters and/or one that is fine- tuned on relevant data) is employed. However, a sufficiently hard dataset should not induce a perfect model performance (i.e. near 100% ac- curacy score). This can be tested with models of different sizes and thus capabilities.
Evidence of Dataset Bias. We start from the NCAQ settings, i.e., only the answer options (a and i) are given as model input, without the ques- tion and video, for both training and evaluation. Under this setting, we also compare models' perfor- mance with different perturbations on the answer options. Table shows that the basic LSTM model outperforms the random guess by 9.45% on v1 (i.e. Social-IQ). With the unrcasonable inputs (with no context nor question), these accuracy scores show that the Social-IQ dataset is biased.
RIWI: Replace i with i', (q. a i) + (g. a. a')
RIWA: Replace i with a', (q, a) + aq)a,a')
RAWI: Replace a with e', (q, a, i) + i.)
RAWA: Replace with a', (g, a, i) + (g. ii
With the above perturbations, we expect the dataset to induce the following robustness be- haviours. With RIWI or RIWA applied to the dev/test set, we should expect that a model's 's per- formance should not significantly deviate from
We postulate that while stronger model (i.e. MPM) should obtain better performance than LSTM, without being given sufficient information,
htti::://ccmmmulttii caw.plaab.bbaaa........... lab. wit.g io/ vocial-iq-2.0

---

Data
Model
Settings
A2
A4
Random
none
50
25
I-didriintted whal Nuaghlse Extoviding
LSTM
NCAQ
59.45
34.84
incomect
NCAQ
T00
T0U
NCAQ+RIWI
97.37
99.97
v1
T5-small
(MPM)
NCAQ+RIWA
50.21
25.03
NCAQ+RAWI
49.93
23.76
NCAQ+RAWA
97.25
100
NCAQ
63.35
NCAQ+RIWI
59.66
T5-small (MPM)
NCAQ+RIWA
24.72
-15
NCAQ+RAWI
-50
NCAQ+RAWA
62.36
Table 3: Model performance A2 (binary choice) and A4 (multiple choice) under different experimental set- tings, in which only answer options given model inputs (but not questions nor context). represents the results are inapplicable.
Figure T-SNE visualization of correct and incorrect answer options. Red dots are incorrect answer options and blue dots are correct answer options.
even the stronger model should not perform unrea- sonably well. Thus, we experiment with T3-small, modestly-sized yet more capable model. As it can be seen in Table 3, T5-small outperforms LSTM by large margin v1. Surprisingly, it also achieves a perfect 100% accuracy score on v1 and 63.35 on v2 without being given the context nor the ques- tion. These results provide strong evidence of the biases in these datasets.
RAWA. It should lead to A2 with 50% and A4 with 25% performance since the correct answer option is replaced with an irrelevant correct an- swer of another question.
Contrary to our intuition. RAWA leads near- perfect performance of 97.25% on A2 and even better 100% on A4.
These unexpectedly high scores indicate that the model can easily distinguish the correct answer options from the incorrect ones of the original dataset, regardless of the question they are used for, consistent with the results of RIWI.
Finally, we study the other four perturbation set- tings by applying them to the dev sets. Below we analyse the performance on vl in detail, followed by discussion v2.
RIWL Similar to the performance on the origi- nal dataset, T5-small achieves an unreasonable performance of 97.37% on A2 and 99.97% in A4, It indicates that the model can easily distinguish the correct answer from the incorrect options.
Figure shows the T-distributed Stochastic Neighbor Embedding (T-SNE) visualization of the embeddings of all answer options in the Social-IQ dev set. We can observe a clear boundary between correct and incorrect answer options. The above re- sults provide compelling evidence of the unwanted bias in Social-IQ, manifested in T5-small's strong capability in distinguishing the correct and incor- rect answer options.
RIWA. It leads to large performance degrada- tion: A2 from 100% to 50.21% 1,, A4 from 100% to 25.03%, similar to random guess (i.e., 50% and 25%). This shows that T5-small is unable to distinguish the correct answer options, regardless of the question it is used for.
Similar evidence can be found in Social-IQ-2.00 as can be seen the v2 rows in Table
RAWL This produces a dataset containing only incorrect answer options. We consider the in- correct answer option that replaces the correct answer option as the correct answer.
DeSIQ: Debiased Social-IQ
In this section, we first describe our approach to debias Social-IQ. We then study the effectiveness of our debiasing approach and the resultant DeSIQ datasets, by comparing the performance of both LSTM and T5-small on DeSIQ in different settings.
Intuitively, it should lead model to randomly guess, none of the options is correct. In Ta- ble 3, we observe that RAWI leads large performance drop: A2 from 100% to 49.93%, A4 from 100% to 23.76%, indicating that T5-small cannot distinguish incorrect answers from each other, confirming our intuition.
3.1 Constructing DeSIQ
We propose the following perturbation-be based ap- proach to debias Social-IQ and construct a more

---

meaningful and challenging dataset on social intel- ligence. Specifically, we apply the RIWA pertur- bation on both the training and development sets of Social-IQ, ie substituting the incorrect answer options with correct answer options from the other questions. We construct two debiased datasets";
9
MLP
MLP.
MLP
m
m,
Backbone
DeSIQa- Given an original triplet (g, a, i), we randomly sample another triplet (q', a'',a,, a', from another video. Thus, for each original triplet a,,,, we form new triplet (g,a,a').
P
MLP
MLP
DeSIQ.. We sample (d','', d','' from the same video for each (g, a, i). Similarly, we replace the incorrect answer option with a'. Since and o' are from the same video, their answers can have a higher chance of referring to the same entity that appears in the video. Thus, DeSIQ, is more challenging dataset of (g, a. a').
Figure 4: Our model structure to address the new bench- mark DeSIQ. The A2 configuration is shown here for illustration purposes. q. a, i, U, 8 are question, correct answer, incorrect answer, transcript, video and audio features; my. Ma. m,, myg My, my are the correspond- ing features after projection the same dimension; and Tg Ta: Vt, T', are the corresponding embeddings after going through the same backbone model. [o] de- notes the concatenation operation and dashed boxes denote optional input features. Three multi-layer per- ceptrons (, MLP, and ML.,,) in yellow above are projectors mapping textual, video and audio features to the same space. The output will be the two scores $1, $2 (four scores for multtple-choice) representing the correct and incorrect answer option respectively.
An example video and some associated questions and answer options for both Social-IQ and DeSIQ; can be seen Figure 1. For Social-IQ-2.0, we do the same approach to obtain 2e00Q&-2.0.
3.2 Effectiveness of the Debiasing Approach We set up a number of models both fully super- vised and zero/few-shot learning settings to show the effectiveness of our debiasing approach,
T5-small, suffers a larger performance drop on both A2 and A4 of DeSIQ, (to 50.16% and 34.15% respectively), although 100% A2 on Social-IQ.
Supervised Learning. We train the LSTM and T5-small on Social-IQ, DeSIQa and DeSIQ, in the same architecture (Figure 2), and train T5-small on Social-IQ-2.0. Table 4, Table and Table 6 show the results, where the relevant results are shaded in gray The second column "Input" represents the input used in both the training and evaluation procedures, where "a", "q", "t", and "v" represent answer options, the question, the transcript and the video, respectively. The third column "Concat" rep- resents different model architectures. The symbol "X" denotes that all inputs are separately encoded as in Figure 2. which is the focus of this subsection. The symbol "v" denotes that all inputs are concate- nated and encoded as one sequence as in Figure 4, which will be discussed in the next section.
The results in Table show that DeSIQ, effec- tively reduces the bias in Social-IQ. For LSTM, DeSIQ, leads to performance drop of 59.45% -> 48.24% on A2 and 34.84% + 27.06%. For T5- small, we can observe performance drop of 100% 48. 73%% on A2 and of 100% + 33.53% A4. Comparing results in Tables and 5, we see DeSTQ, is generally more challenging than DeSIQa. i.e. compare T5-small's performance un- der the same settings across the two datasets. For instance, with the question feature added ("q+a" inputs), T5-small achieves 60.55% DeSIQa and 49.17% on DeSIQ, in A2, and 27.57% on DeSIQa and 24.22% on DeSIQ, A4.
As seen in Table 4, DeSIQu largely reduces the bias in Social-IQ, effectively reducing the perfor- mance of both LSTM and T5-small close to ran- dom guess. For LSTM, when given only answer op- tions, we observe a performance drop 59.45% >> 48.52% on A2 and 34. 84.84% + 27 27.23% on A4. For
In Table 6, DeSIQ also reduces the bias in Social-IQ-2.0, effectively reducing the perfor- mance of T5-small close to random guess. For T5-small, it suffers a larger performance drop on A4 of DeSIQ.-2.0 (63.35% to 28.07%).
Zero-shot and Few-shot Learning. We employ the strong GPT-3 model (Brown et al., 2020) and ChatGPT (OpenAI, 2022) with both zero-shot and
'Below we describe the dataset construction process for the A2 configuration. Similar perturbations are applied to the A4 configuration of Social-IQ.

---

A2
A4
Dataset
Input
Concat
LSTM
T5-small
seaaaallpepph
LSTM
T5-small
T5 DessmallDetph
59.45
100
100
34,84
100
100
Social-IQ
qta
59.78
100
100
38.55
100
100
qta+t
X
60.00
100
100
43.84
100
100
q+a+v
64.38
100
100
46.08
100
100
X
48.52
50.16
50.33
27.23
34.15
28.97
DeSIQa
q+a
X
58.58
60.55
50.19
26.05
27.57
25.78
qta+t
60.46
50.16
50.40
27.59
28.84
27.15
q+a+y
61.05
50.59
$0.70
25.91
27.60
25.55
49.52
50.33
36,30
38.18
DeSIQa
q+a q+a+t q+a+v
73.47
75.69
62.43
72.91
13.17
74.69
76.77
29.22
70.80
74.51
56.67
76.72
74.99
41.77
72.69
73.24
Table 4: Accuracy on the Social-IQ and DeSIQ, development sets. Results shaded in gray are relevant to Sec. 3.
A2
A4
Dataset
Input
Concat
LSTM
T5-small
LSTM
T5-small
T5-smallnetphs
X
39.45
100
100
34.84
100
100
Social-IQ
q+a
X
59.78
100
100
38.55
100
100
q+a+1
x
60
100
100
43.84
100
100
q+a+v
x
64.38
100
100
46.08
100
100
X
48.24
48.73
48.96
27.06
33.53
28.12
DeSIQ.
q+a
X
59.97
49.17
$9.59
26.16
24.22
25.20
q+a+1
X
60.02
58.89
56.83
27.31
26.79
23.83
q+a+v
X
61,00
59.19
56.42
26.99
25,00
24.22
V
48.24
48.73
48.71
29.67
29.17
32.32
DeSIQ,
q+a
59.35
63.08
63.42
34.73
62.47
60.58
q+a+1 Q+a+v
11.52
65.41
67.70
22.33
65.23
51.69
51.04
65.96
65.02
32.56
56.61
55.05
Table 5: Accuracy on the Social-IQ and DeSIQ, development sets. Results shaded in gray are relevant to Sec.
Dataset
Input
Concat
A4
the two answer options is randomly shuffled. The zero-shot prompt is constructed as follows:
a
x
63.35
Social-IQ-2.0
q+a
x
64.63
q+a+t
x
64.06
"Choose the correct answer option corresponding to the question:
q+a+v
x
62.28
X 28.07
+ + A: a " "+i
DeS0Qa-2.0
q+a
x
28.45
q+a+t
X 22.17
For the few-shot evaluation, we use the question similarity to find exemplars for in-context learn- ing. For each question in the development set, we choose the top-3 most similar questions from the training set. We measure the semantic distance between questions based on their embeddings ob- tained from
q+a+v
X
24.13
q+a+s
X
25.87
V
28.07
q+a
V
57.23
De00Qu
q+a+t
V
52.02
q+a+v
/
68.93
q+a+s /
74.13
q+a4tav4s
/
37.72
(Reimers and Gurevych, 2019). The few-shot prompts leverage the same format as in the zero-shot evaluation, with the correct option appended to each exemplar:
Table Accuracy on the Social-IQ-2.0 and DeSIQ.-2.0
development sets. Results shaded in gray are relevant to Sec. 3.
few-shot learning to show the strength of our de- biased datasets. Social-IQ experiments are per- formed in the A2 configuration using GPT-3, while Social-IQ-2.0 experiments in the A4 using Chat- GPT. For zero-shot evaluation, we concatenate the question with correct and incorrect answer options (i.e. "q+a") to form the prompt', where the order of
"Choose the correct answer option corresponding to the question: (q' + " A: "+ a' + B: "
or B*) zero-shot prompt
Table 7 shows the results. For Social-IQ, under the zero-shot setting, GPT-3 can obtain 58.26% with "q+a" and 64.63% with "q+a+t" on Social-IQ. In comparison, under either zero-shot or few-shot setting, both the DeSIQ, and DeSIQ, dataset lead
WWe observe that when only given answer options hut not the question, GPT-3 tends to select the first option.

---

Dataset
Input
AGPTT-3VA (ChaiGPT)
T5-small on these corpora and then finetune it on the downstream Social-IQ and DeSIQ datasets. We call this variant TS-small Delphi-
Zero-shot
shor
Social-IQ
q+a
58.26
56.22
q+a+t
64.63
DeSIQu
q+a
qta+l
54.78
59.79
54.13
4.1 Results on DeSIQ
DeSIQ.
q+a
54.39
53.29
We analyze the effectiveness of our proposed archi- tecture, and the effect of the distillation of common- sense knowledge. The results of our new model architecture are shown in the bottom portions of Tables 4 and 5, where the inputs are concatenated ("V" for the column "Concat"). We can make the following observations:
q+ā++ 60.13
q+a 59.61
58.02
DeSIQu-2.0
q+a+t 59.24
Table 7: GPT-3 performance on the A2 of Social-IQ and DeSIQ, ChatGPT performance on the A4 of DeSIQ-2.0.
to a performance drop of more than 4%. Under the few-shot setting for "qta", GPT-3 does not seem to learn shortcuts, as the performance is unchanged compared to the zero-shot setting². These results
Both T5-small and De-small outperform the LSTM baseline both DeSIQa and DeSIQ, while not achieving near perfect accuracy, show- ing the effectiveness of our proposed architecture as well as the unbiased nature of DeSIQ.
show that DeSIQa and DeSIQ, are less biased and more challenging than Social-IQ. For Social-TQ- 2.0, the performance does not change that much when leveraging ChatGPT under both zero shot and few-shot learning settings, which also proves it is less biased than Social-IQ.
When the question is given as part of the model input, T5-small and pels (V) signifi- cantly outperform the vanilla versions (X), show- ing the effectiveness of our model architecture.
4 Setting Baseline Performance on DeSIQ
Injecting commonsense knowledge can indeed improve model performance on social intel- ligence. T5-small with "q+a+t" inputs shows the best A2 score 76.77% and A4 and 74.51% on DeSIQa, and 67.70% in A2 on DeSIQ,. On DeSIQá. it outperforms T5-small all but one settings (q+a+v for A2). On DeSIQ,. however, T5-small shows competitive performance, and significantly outperforms T5- small on A4 for both qta+t. We leave the investigation of this result to future work.
For our more challenging DeSIQ benchmark, we introduce new baseline model to better handle multi-modal inputs. Its architecture is shown in Figure 4. Compared with the model in Figure 2, we add three more projection layers (three yellow MLPs) to map the original feature representations into the same dimensions. We then concatenate all the resulting representations as the inputs to a back- bone MPM. For DeS1Q& containing raw data, we employ Vision Transformer (ViT) (Dosovitskiy et al., 2021) and Wav2Vec 2.0 (Baevski al., 2020) obtain the video and audio representations re- spectively. We note again that raw video and au- dio files are not available for v1, thus we develop the above architecture to uniformly handle both datasets, and leave how to best use multi-modal inputs in DeSIQ-2.0 2.0 for future work.
In many cases, adding the transcript can help improve model performance, and usually more effective than adding the video modality. Since TS-small is pretrained on a textual corpus, it is reasonable that adding the video modality may decrease model performance.
Compared to DeSIQa, DeSTQ, is a more chal- lenging dataset, as except for "a", performance of Tmlll and T5 smalmall drops for all others.
As social intelligence usually requires com- monsense knowledge, we posit that injecting commonsense knowledge into the backbone lan- guage model in our architecture would improve the model's performance. Therefore, inspired by Jiang et al. (2021), we distill commonsense so- cial knowledge from the following datasets into T5- small: Social Chemistry 101 (Forbes et al., 2020), ETHICS (Hendryeks et al., 2021) and Moral Sto- ries (Emelin et al., 2021). Specifically, we pretrain
Comparing the performance of q+a+t/++a+v and q+a, we can observe that both T5-small and Deishall can learn some shortcuts, they achieve comparable performance when only given the question and answers as input.
Some examples are shown in Appendix A Figure 5, illustrating the influence of different modalities.
The first two examples show how the transcript and video features may provide clues for answering the
*We could not perform few-shot experiments with "qua+t" due to GPT-3's limit of prompt length.

---

question. For instance, the first example cannot be correctly answered based on "q+a", since the transcript contains the required information. T5- pelphi is the only model that predicts correct options for the last example in Figure 5, which we attribute to Delphi's commonsense knowledge.
5 Related Work
Debiasing. Shah et al. (2020) proposed number of expectations examine a model's performance on a number of multiple-choice QA datasets and observed that the model (RoBERTa) falls short of the expectations. Different from this work, we establish a systematic methodology, consisting of six novel methods, to examine dataset. And we design some experimental settings on both Social- IQ and Social-IQ-2.0..
4.2 Results on DeSIQ-2.0
For DeSIQ-2.0, we can apply multi-modal model using the raw videos and audios. The experimental results are in Table 6. Apart from some similar observations DeSIQ-1.0 above, some new con- clusions can be made follows:
Language Dependence/Prior is actually a MODEL side bias resulting in the model largely depending on one major modality (usually text). Reducing it can be regarded as an optimization problem. Gat et al. (2020) try to balance the in- fluence of text and image from the MODEL side. Though the paper includes Social-IQ dataset and gets positive results, it doesn't 't realise the bias's existence in the original Social-IQ dataset.
Adding audios or videos can help improve model performance. Moreover, audios are more effec- tive as the model achieves overall best A4 score of 74.13% under the "q+a+s" setting.
Employing raw transcripts can reduce model per- formance (57.23% + 52.02% as they are usu- ally times longer than other input features in length, which can largely influence the represen- tation learning procedure of other inputs.
Shortcut a DATA side bias resulting in the model easily learning the pattern/repeated word in one dataset. For example, some keywords can occur both in the question and the correct answer, but not in the incorrect answers, so that the model directly gets clues from this overlap. Ye and Ko- vashka (2021) identify the shortcut and show its negative effects. However, they only modify the validation data and propose a masking approach to perform more robust training on the MODEL side.
Compared with ChatGPT in Table 7, our best result outperforms 24.52% A4, which shows DeSIQ-2.0 be a challenging dataset.
We also conduct experiments with settings "att" and "a+v", but don't include them the paper. Af- ter debiasing, both settings for the proposed model on DeSIQ2.0 are near the random guess perfor- mance: "att" 22.66% and "atv" 26.46%. Thus, questions are necessary when compared with the performance of "qta+t" and "q+a+v" inputs in Ta- ble 6.
In this paper, we start from the DATA side and also peform debiasing on the DATA side. Moreover, the bias we identify in the Social-IQ dataset is not the same kind, which is mainly in the answers and much harder be debiased in the DATA side. Thus, though they share some similarities, we consider it a new task.
4.3 Further Research Questions
The above results show the lack of biases and challenging nature of our DeSIQ datasets as well as promising performance by modestly-sized lan- guage models. These results lead to the following important research directions for further investiga- tion:
Multi-modal Question Answering. With dif- ferent multiple input modalities, such as image and video, mudti-modal question answering prob- lem is more challenging and has been rising more and more attention in the past few years, Datasets like MovieQA (Tapaswi al., 2016), TGIF-QA (Jang et al., 2017), TVQA (Lei et al., 2018) and TVQA+ (Lei al., 2020) provide im- ages, GIFs or video clips in addition to text-based single-tun -urle-turn questions. There are some datasets like AVSD (Alamri et al., 2019) that require dialogue history to predict answers for multi-turn questions. All these datasets evaluate model capacity of per- ceive the contextual information contained in both text and non-text modalities.
Are there still noticeable biases in DeSIQ, and if so, how to further debias it?
What is the performance of stronger language models on DeSIQ?
How to effectively incorporate socio-cultural and commonsense knowledge into large language models for this task?
How to utilize multi-modal language models to better exploit video and audio input?

---

Social Intelligence Learning. Understanding and reasoning about social commonsense knowl- edge and human interactions essential for cog- nitive social intelligence learning. Bosselut et al. (2019) present a comprehensive study on automatic commonsense knowledge base construction, which mines the intents and reasons behind human behav- iors. (Jiang al., 2021) propose commonsense moral model to better understand social norms and make reliable ethical judgments on world hu- man actions. In this paper, we focus on the Social- IQ dataset (Zadeh et al., 2019), a benchmark pro- vides a diverse annotated set of videos and question- answer pairs. We run all the experiments on this dataset because it is much more related to social intelligence learning than other datasets.
Ethics Statement
Although the benchmark is designed for studying human behaviors and research purposes only, the resources and findings could be used unexpectedly. For example, it is possible that harmful content exists in the Social-IQ dataset, thus also in our DeSIQ datasets, based on which trainable models could turn from positive to a negative perspective. Thus, is prudent for researchers working on social intelligence to pledge to only make ethical use of our benchmark datasets.
A Acknowledgement
This material is based on research partially sponsored by the DARPA Assured Neuro Sym- bolic Learning and Reasoning (ANSR) program under award number FA8750-23-2-10116 the DARPA Knowledge Management Scale and Speed (KMASS) program under award number HR001 12220047, and the DARPA Computational Cultural Understanding (CCU) program under the agreement number HR001 122C0029. The U.S. Government is authorised to reproduce and dis- tribute reprints for Governmental purposes notwith- standing any copyright notation thereon. The au- thors are grateful to the anonymous reviewers for their helpful comments.
6 Conclusion
Social intelligence is an essential ingredient for effective human-computer communications. In this paper, we analyze Social-IQ, multiple-choice question answering benchmark dataset for social in- telligence. Our empirical analysis reveal the severe biases present in Social-IQ, which can be casily exploited by modestly-sizer language models such T5-small to achieve perfect accuracy on its de- velopment set. We construct the DeSIQ benchmark by applying simple perturbation-base techniques on Social-IQ and show that the DeSIQ vastly re- duce the biases Social-IQ. Moreover, we pro- pose a new model architecture on DeSIQ and set strong performance baselines for this challenging new dataset. Finally, our comprehensive analyses open up number of important research questions for further investigation.
References
Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K. Marks, Chiori Hori, Peter Anderson, Stefan Lee, and Devi Parikh. 2019. Audio visual scene- aware dialog. In IEEE Conference Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 7558- 7567. Computer Vision Foundation / IEEE.
Computer
Limitations
Eleni Andreou. 2006. Social preference, perceived pop- ularity and social intelligence: Relations to overt and relational aggression. School Psychology Interna- tional, 27(3):399-33 351.
For the proposed model architecture designed to address the new DeSIQ benchmark, we mainly em- ploy text-based language models and pretrain them on text-based corpora. The exploration of power- ful multi-modal language models, instead of using the projection function as is done in this paper, is thus an important future research work direction. Due to resource constraints, all the experiments in this work were under conducted only once with the same random seed equals 42. Multiple runs with different random seeds would enable us perfor- mance statistical significance tests of the results, and thus make the findings more reliable.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural information Processing Sys- tems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: commonsense transformers for auto- matic knowledge graph construction. In Proceedings

---

of the 57th Conference of the Association for Compu- tational Linguistics, ACL 2019. Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4762-4779. Association for Computational Linguis- tics.
Yunseok Jang, Yale Song. Youngjae Yu, Youngjin Kim. and Gunhee Kim. 2017. TGIF-QA: toward spatio- temporal reasoning in visual question answering. In 2017 IEEE Conference on Computer Vision and Pat- lern Recognition, CVPR 2017. Honolulu, HI, USA, July 21-26, 2017, pages 1359-1367. IEER Computer Society.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger. Tom Henighan. Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Matcusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πly Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12. 2020, virtual.
Robin Jia and Percy Liang. 2017. Adversarial exam- ples for evaluating reading comprehension systems, In Proceedings of the 2017 Conference Empiri- cal Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ro- nan Ie Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. 2021. Delphi: Towards machine ethics and norms. CoRR, abs/21 10.07574.
Yichen Jiang and Mohit Bansal, 2019. Avoiding reason- ing shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 2726-27 2736, Flo- rence, Italy. Association for Computational Linguis- tics.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. 2018. TVQA: localized, compositional video ques- tion answering. In Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Pro- cessing, Brussels, Belgium, October 31 - November 4, 2018, pages 1379-1379. Association for Computa- tional Linguistics.
Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. 2021. Moral stories: Situated reasoning about norms, intents, actions, and their con- sequences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pages 698-718. Association for Computational Linguistics.
Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal. 2020. TVQA+: spatio-temporal grounding for video question answering. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8211-8225. Association for Computational Linguistics.
Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chem- istry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653-670, Online. Association for Computational Linguistics.
OpenAL. 2022. Chatgpt: Optimizing language models for dialogue. Accessed 2023-01-13.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn, Res., 21:140:1-140:61
Itai Gat, Idan Schwartz, Alexander G. Schwing, and Tamir Hazan. 2020. Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies. In Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020. December 6- 12, 2020, virtual.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert Sentence embeddings using siamese nett-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Daniel Goleman. 2007. Social intelligence. Random house.
Dan Hendrycks. Collin Burns, Steven Basart, Andrew Critch, Jerry Li. Dawn Song, and Jacob Steinhardt. 2021. Aligning Al with shared human values. In 9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Krunal Shah, Nitish Gupta, and Dan Roth. 2020. What do we expect from multiple-choice QA systems? In Findings of the Association for Computational Lin- guistics: EMNLP 2020, Online Event, 16-20 Novem- ber 2020, volume EMNLP 2020 of Findings of ACL.

---

pages 3547-3553. Association for Computational Linguistics.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. Movieqa: Understanding stories in movies through question-answerrigg In 2016 IEEE Confer- ence Computer Vision and Pattern Recognition, CVPR 2016, Vegas, NV, USA, June 27-30, 2016, pages 4631-4640, IEEE Computer Society.
Edward Thomdike. 1920. Intelligence and its uses. Harper's magazine.
Keren Ye and Adriana Kovashka. 2021. A study of the shortcut effects visual commonsense reasoning.
Proceedings of the AAAI Conference on Artificial Intelligence, 35(4))3181-318)
Amir Zadeh, Michael Chan, Paul Pu Liang. Edmund Tong, and Louis-Phillippe Morency. 2019. Social- iq A question answering benchmark for artificial social intelligence. In IEEE Conference Com- puter Vision and Patern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8807- 8817. Computer Vision Foundation IEEE.
A Appendix
A.1 Experimental Settings
For all experiments, we fix our random seed at 42. We run fully supervised learning one A40 GPU with 40GB memory, and set the learning rate as le-4 as well as early stopping by monitoring the loss on the development set. Typically, takes 2-3 hours using only features to finish 100-epoch training with a batch size 8.
We also employ GPT-3 (175B parameters) as it is the current state-of-the-art language model in variety of NLP tasks for in-context learning For the few-shot evaluation, select top-3 most similar questions from the training set. We did not other choice of due to both budget and time constraints, which we leave for future work.

---

Figure 5: Predictions of different models on DeSIQa benchmark. We show the answers, question. transcript and video modalities in the first four columns, and predictions of different models using our settings of input features. Red ellipses are predictions based on certain features. Some explainable clues shaded yellow
a q+a+t q
a qta qtat q A
qtaH g
B
A
B
B
B
B
B
Prediction
A
A
A
B
A
q+a
B
B
B
A
B
B
B
B
B
A
A
B
B
B
T5-small Delphi
T5 75-small Delphi
T5 r5-small
LSTM
LSTM
LSTM
Delphi
Video
Socad SocailChemn (Chem101 clues (DELPHI pretraining corpus): it's fun to be happy about getting a high score.
just don't want my children to grow up with the same experience had.
-that's what it sounds like when they
split from an evolutionary lineage
And think she's really softened my views on a lot of things.
Are your parents religious?
-Are you captivated yer? So gona to shuftle it
Transcript
-yeah you get 407 points -Oh
They are religious.
Why is the woman upset when she receives her score?
Is the man serious when he asks "are you captivated yet"??
Why does the blonde man seem sad??
Question
Transcript clues: ye Lowlow backg round words.
Video c clues: another person are smi smiling Ling.
(correct); He's upset the brunette man doesn't want his children raised with religion. (incorret): He's sad that he doesn't get along with the brumette man.
A (incorrect): yes, he is being serious
(correct): It is not as high as she thought it would be B (incorrect): She is just acting upset trick the opponent
B (correct): no, he being
sarcastic for comedic affect
Answer


Reviews:
[{"review_id": "SLC24CVnSv", "reviewer": "Reviewer_hZcY", "summary": "", "questions": "1)\tIn the proposed benchmark DeSIQ, in order to debias the answers, the incorrect answers are substituted with correct answers of other questions. However, I wonder that whether the questions are still needed for models to achieve an excellent score, since the problem is transformed into a matching problem between answer and video/transcript, \n\n2)\tI also noticed that there were no experiment with the input of “a+t/v”.\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: DeSIQ’s data are totally from the existing dataset Social-IQ without expansion or further cleaning, which means that the qualities of those data are completely dependent on Social-IQ.\n\nA1: We’d like to emphasise that our aim is indeed to improve the quality of Social-IQ by debiasing it. While paraphrasing questions/answers could produce more training data, doing so may introduce other biases. Therefore, we will leave further cleaning and expansion to future research.\n\nQ2: As mentioned in the paper, all experiments were conducted only once, which makes the results less convincing.\n\nA2: Though we did our experiments only once, we chose the same random seed, ran 2-3 times for each setting, and got the same value on the accuracy score. If time and resources permit, we can run all the experiments with different random settings.\n\nQ3: In the proposed benchmark DeSIQ, in order to debias the answers, the incorrect answers are substituted with correct answers of other questions. However, I wonder whether the questions are still needed for models to achieve an excellent score, since the problem is transformed into a matching problem between answer and video/transcript.\n\nA3: The questions are indeed needed since the model should know which answer it needs to generate/match. Please see the answer of Q4 for a detailed explanation and empirical evidence.\n\nQ4: I also noticed that there was no experiment with the input of “a+t/v”.\n\nA4: We did conduct  experiments with settings “a+t” and “a+v”, but didn’t include them in the paper. After debiasing, both settings for the proposed model on DeSIQ2.0 are near the random guess performance: “a+t” 22.66% and “a+v” 26.46%. Thus, questions are necessary when compared with the performance of “q+a+t” and “q+a+v” inputs in Table 6. If applicable, we could involve these results in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1)\tThis paper defined a comprehensive methodology to identify bias in multiple choice question datasets. This is an interesting findings for the related study.\n\n2)\tThe authors proposed to construct a unbiased challenging benchmark DeSIQ in the field of Social Intelligence Understanding.\n\n3)\tThe authors conducted extensive experiments to prove its effectiveness, and proposed a new model architecture on DeSIQ along with baselines.\n", "reasons_to_reject": "1)\tDeSIQ’s data are totally from the existing dataset Social-IQ without expansion or furthur cleaning, which means that the qualities of those data are completely depend on Social-IQ. \n\n2)\tAs mentioned in the paper, all experiments were conducted only once, which makes the results less convincing. \n", "questions_to_authors": "1)\tIn the proposed benchmark DeSIQ, in order to debias the answers, the incorrect answers are substituted with correct answers of other questions. However, I wonder that whether the questions are still needed for models to achieve an excellent score, since the problem is transformed into a matching problem between answer and video/transcript, \n\n2)\tI also noticed that there were no experiment with the input of “a+t/v”.\n", "paper_topic_and_main_contributions": "This paper defined a comprehensive methodology to identify the bias in Social-IQ, a dataset of multiple choice questions on videos of complex social interaction, and constructed a new unbiased and more challenging benchmark named DeSIQ. Experiments on both supervised learning and zero/few-shot learning were carried to prove the effectiveness of the Debiasing Approach. Besides, a new model architecture along with baselines on DeSIQ was proposed. In conclusion, the paper introduced the related work and discussed important research directions for further investigation.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SWyftKBHZt", "reviewer": "Reviewer_tCWT", "summary": "", "questions": "The construction of unbiased, challenging datasets for social intelligence, like DeSIQ, requires considerable effort and a comprehensive understanding of the biases in existing datasets. What measures have the authors taken for that? Did they use any well-structured documents to construct guidelines?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: The construction of unbiased, challenging datasets for social intelligence, like DeSIQ, requires considerable effort and a comprehensive understanding of the biases in existing datasets. What measures have the authors taken for that? Did they use any well-structured documents to construct guidelines?\n\nA: The identification of biases is based on our comprehensive empirical evaluation of the original dataset, on which a small model (i.e. T5-small) can achieve 100% accuracy with answer choices as the only input. \nFor debiasing, we employed existing techniques in the literature, but found that the simple perturbation approach we proposed is the most effective. We will add more discussions on the debiasing approaches we experimented in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "By identifying and rectifying biases, new, more challenging datasets (like DeSIQ) can be constructed for better benchmarking of AI's social intelligence capabilities. The studies can shed light on the effect of different aspects of machine learning models (like model size, model style, learning settings, common-sense knowledge, and multi-modality) on performance.", "reasons_to_reject": "The construction of unbiased, challenging datasets for social intelligence, like DeSIQ, requires considerable effort and a comprehensive understanding of the biases in existing datasets. What measures have the authors taken for that? Did they use any well-structured documents to construct guidelines?", "questions_to_authors": "The construction of unbiased, challenging datasets for social intelligence, like DeSIQ, requires considerable effort and a comprehensive understanding of the biases in existing datasets. What measures have the authors taken for that? Did they use any well-structured documents to construct guidelines?", "paper_topic_and_main_contributions": "This paper presents a deep analysis of the Social-IQ dataset, a benchmark for studying social intelligence, as well as introduces a new dataset called DeSIQ. The authors identified substantial biases in the Social-IQ dataset which can be exploited by moderately strong language models to achieve perfect performance without even needing the context or the question.\n\nThe Social-IQ dataset was developed to evaluate social intelligence in AI models. It is a multiple-choice question answering (MCQA) dataset with questions, answer options, videos, and other multi-modal inputs. However, the study found that even relatively simple models could achieve 100% accuracy due to the biases present in the dataset. This led the authors to develop a new dataset, DeSIQ, using simple perturbations to the Social-IQ dataset to significantly reduce these biases.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ouPSOAhukP", "reviewer": "Reviewer_BsAc", "summary": "", "questions": "The prompt used for the zero-shot experiments can be crafted in different ways. The context in the prompt could yield different results. Have you tried different prompts for these experiments (and maybe observed different results based on the context)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Previous work (Removing Bias by Gat et al.) have focused on techniques to diminish the bias at the model level. The manipulation of the data has also been explored by Ye et al. ( Shortcut Effects in Visual Commonsense Reasoning). Although previous work explains techniques to manipulate the data/model, it does not focus on the specific DeSIQ dataset. Some overlapping concepts could be a weakness of the work.\n\nA1: Thank you for pointing out these papers. Below we summarise the main differences of our work from these papers.\n\nLanguage Dependence/Prior is actually a MODEL side bias resulting in the model largely depending on one major modality (usually text). Reducing it can be regarded as an optimization problem. Gat et al. (Removing Bias in Multi-modal Classifiers) try to balance the influence of text and image from the MODEL side. Though the paper includes Social-IQ dataset and gets positive results, it doesn’t realise the bias's existence in the original Social-IQ dataset.\n\nShortcut is a DATA side bias resulting in the model easily learning the pattern/repeated word in one dataset. For example, some keywords can occur both in the question and the correct answer, but not in the incorrect answers, so that the model directly gets clues from this overlap. Ye et al. (Shortcut Effects in Visual Commonsense Reasoning) identify the shortcut and show its negative effects. However, they only modify the validation data and propose a masking approach to perform more robust training on the MODEL side.\n\nIn our paper, we start from the DATA side and also peform debiasing on the DATA side. Moreover, the bias we identify in the Social-IQ dataset is not the same kind, which is mainly in the answers and much harder to be debiased in the DATA side. Thus, though they share some similarities, we consider it a new task.\nWe will include these discussions in our final version.\n\nQ2: The prompt used for the zero-shot experiments can be crafted in different ways. The context in the prompt could yield different results. Have you tried different prompts for these experiments (and maybe observed different results based on the context)?\n\nA2: Yes, the context design could largely influence the output quality. Unfortunately, we didn’t try other prompts and only used the prompts listed in our paper due to time and resource limitations. As zero-shot/few-shot learning is not the focus of this work,  we leave prompt optimization to future research work.\n\nQ3: The tables on page 6 contain shaded results to illustrate relevant sections in the paper. While this is useful, it could also confuse readers who glance through the tables first.\n\nA3: Thank you for mentioning that. We will explore other presentation options to improve their readability in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper delivers a comprehensive analysis on different architectures and their biases when tested on DeSIQ data. \nThe new dataset also demonstrates how accuracy fluctuates when the same model is used, and how shortcuts can be learned when similar context is used for the incorrect answers. \nThis work is also well written and the experiments are sound. The ablation experiments also illustrates in different ways the biases found in the data. ", "reasons_to_reject": "Previous work (Removing Bias by Gat et al.) have focused on techniques to diminish the bias at the model level. The manipulation of the data has also been explored by Ye et al. ( Shortcut Effects in Visual Commonsense Reasoning). Although previous work explain techniques to manipulate the data/model, it does not focus on the specific DeSIQ dataset. Some overlapping concepts could be a weakness of the work ", "questions_to_authors": "The prompt used for the zero-shot experiments can be crafted in different ways. The context in the prompt could yield different results. Have you tried different prompts for these experiments (and maybe observed different results based on the context)?", "paper_topic_and_main_contributions": "This paper analyzes and reveals biases found In the Social-IQ dataset. The insights are used to generate a new dataset (DeSIQ) that reduces biases the biases seen in the initial dataset. This datasets are tested on different architectures such as T5-Small, GPT-3, etc. \n\nDifferent configurations for the question-answer setting are tested such as: the replacement of incorrect answers, replacement of context, replacement of the correct answer, etc. The different model sizes are also tested to see how model-size affects the capability to learn/obtain a specific bias that does not generalize.\n\nExperiments where hypothesis are replaces with other correct answers from the set (RIWA) shows large degradations in accuracy, and potential biases learned by these models. \n\nThe new DeSIQ dataset is also generated with different settings/contexts in mind to generate a more challenging set. Context from the same or different videos can be used to create different mixtures of the data. \n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The tables on page 6 contain shaded results to illustrate relevant sections in the paper. While this is useful, it could also confuse readers who glance through the tables first. \n\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a comprehensive method to analyze biases in Social-IQ dataset, and proposed a new dataset DeSIQ using a simple yet effective perturbation method from which biases are reduced. 

Reviewers appreciated the importance of the problem space, the well-structured content, and solid experiments. They raised two concerns: 1) The lack of diversity in the data construction method: The perturbation method represents just one application of existing debiasing methods; and 2) The lack of generalizability of the proposed dataset: DeSIQ derives from only Social-IQ.
Decision: Accept-Main