Perceptual Score: What Data Modalities Does Your Model Perceive?
Year: 2022

Abstract:
Machine learning advances in the last decade have relied significantly on largescale datasets that continue to grow in size. Increasingly, those datasets also contain different data modalities. However, large multi-modal datasets are hard to annotate, and annotations may contain biases that we are often unaware of. Deep-net-based classifiers, in turn, are prone to exploit those biases and to find shortcuts. To study and quantify this concern, we introduce the perceptual score, a metric that assesses the degree to which a model relies on the different subsets of the input features, i.e., modalities. Using the perceptual score, we find a surprisingly consistent trend across four popular datasets: recent, more accurate state-of-the-art multi-modal models for visual question-answering or visual dialog tend to perceive the visual data less than their predecessors. This trend is concerning as answers are hence increasingly inferred from textual cues only. Using the perceptual score also helps to analyze model biases by decomposing the score into data subset contributions. We hope to spur a discussion on the perceptiveness of multi-modal models and also hope to encourage the community working on multi-modal classifiers to start quantifying perceptiveness via the proposed perceptual score.

heading: 1 Introduction
text: Machine learning advances over the last decade are remarkable. Challenges that seemed daunting merely ten years ago are now a breeze, and new applications that we barely dared to dream about seem achievable within the next few years. Indeed, accuracy metrics on tasks like visual question answering and reasoning suggest significant improvements.
Reported improvements are to a large extent due to the availability of large datasets [1–3], computational performance advances, e.g., for GPUs, and a better understanding about how to encode inductive biases into deep-nets, e.g., by using rectified linear units [4], normalization [5], skip connections [6], transformers [7], etc. However, importantly, developed deep-net architectures are not guaranteed to solve a given task. There is a chance that they may instead exploit dataset biases.
This concern is surely in part due to non-robust training techniques, and a plethora of methods improve classifier robustness [8–10]. However, datasets play an important role in controlling the extracted bias as well. For instance, if correct answers in a question-answering task are significantly shorter than incorrect ones, classifier training should not use answer length as a cue. Although this seems reasonable, for audio-visual scene aware dialog, Schwartz et al. [11] find for example that in many cases the question alone is sufficient to generate a scene-aware dialog response, avoiding the need to look at the video. Hence, in order to assess the suitability of a classifier, we need to understand how much it relies on different data modalities.
To quantify how much a classifier relies on its different input modalities, we introduce the perceptual score. The perceptual score assesses the degree to which a model relies on a modality. To do so the perceptual score permutes the features of a modality across samples in the test set after the classifier
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
How many animals  are pictured?  
What color is the car? Why are the men jumping?   What color is the ground?
Im ag
e Pe
rc ep
tu al
S co re Q ue st io n Pe rc ep tu al S co re
Permute Image
Permute Question
How many animals  are pictured?  
Original Data
How many animals  are pictured?  
Permuted Test
(a) Visual question answering (VQA) data. Modality perceptiveness is measured through a permutation test.
LMH (2019) RMFE (2020) CSS (2020)
40
50
60
70
80 Accuracy
Image Perceptual Score
Question Perceptual Score
(b) Progress of VQA-CP models.
Figure 1: Multi-modal datasets often have undesired biases: (a) To identify those biases we suggest the perceptual score as a new metric. It assesses the change in prediction when a model’s input for some modalities is permuted during testing. If the classifier output remains identical despite permutation, a model doesn’t perceive the modality. (b) Using the perceptual score we identify that recent progress of VQA models may not be entirely due to better reasoning.
was trained, as illustrated in Fig. 1a. If the classifier’s performance drops to or below chance level, the perceptual score is high. This intuitively applies to single-modality models too: randomly permuting test data and labels after training results in chance-level classification accuracy.
Using the perceptual score, we find a surprisingly consistent trend across five popular datasets (VQA, VQA-CP, VisDial, SocialIQ, DISCO): recent, more accurate state-of-the-art multi-modal models for visual question-answering or visual dialog tend to perceive the visual data less than their predecessors (see Fig. 1b). This trend is concerning as answers are hence increasingly inferred from textual cues only. Using the perceptual score also helps to analyze model biases by decomposing the score into data subset contributions. For example, the perception of an image and question varies depending on the question type. None of the recent VQA-CP models showed high image perception scores for ‘number’-type questions. A surprisingly low image perception score is obtained for the state-of-the-art model when confronted with ‘yes/no’ questions.
We hope the perceptual score spurs a discussion regarding the perceptiveness of multi-modal models and we also hope to encourage the community working on multi-modal classifiers to start quantifying perceptiveness of models.
Our contributions:
• We propose the perceptual score, a simple yet effective method for assessing the perceptiveness of multi-modal models towards a modality.
• Our experiments span multiple datasets and models. We find that multi-modal models tend to ignore some modalities while taking shortcuts.
• Subsequently, we investigate the sources of bias on popular multi-modal datasets such as VQA-CP and SocialIQ: We find that SocialIQ is biased by sentiment, and bias in the VQA-CP model results from shifting the training priors to more closely resemble those in the test set.

heading: 2 Related Work
text: The perceptual score assesses the degree to which a classifier relies on a particular input modality. This is related to studying datasets and their biases, methods which aim to reduce the biases captured by classifiers and work which studies the importance of features. We review all three areas next.
Datasets and bias: Data has been a central element for machine learning progress [12–14] in the last two to three decades. The ImageNet challenge [2] and the development of AlexNet [15] sparked the deep learning era. But as datasets grow, biases emerge which may go undetected for a long time. For instance, the background in ImageNet can reveal information about the object class [16]. Also, with the increasing popularity of crowdsourcing systems like Amazon Mechanical Turk, many datasets
are annotated in uncontrolled environments. Different annotators are hence injecting unknown socio-economic properties into dataset annotations [17]. Those dataset biases can be detrimental to the considered task. For instance, meticulously collected visual question answering (VQA) data [1] aims to provide a platform for exciting research to advance image-language understanding. However, it is non-trivial to remove biases from this type of data. Indeed, it was reported that the question solely is sufficient to detect the correct answer [18, 19], i.e., no image information is required. In an attempt to fix this bias, the dataset was re-annotated [20], or the train and test split were reorganized [21]. Similarly, Schwartz et al. [11] reported that in Audio-visual-scene-aware dialog (AVSD) [22], the question cue is often stronger, making the desired video and sound reasoning implausible or unnecessary. Likewise, SNLI [23], aims to determine the correctness of a hypothesis given a premise. However, Gururangan et al. [24] point out that an internal bias exists: linguistic features not related to the premise correlate with the label. Further, the Story Cloze [25] dataset permits to develop models which estimate the correct ending of a story. Schwartz et al. [26] show that, for this dataset, length alone is a powerful feature to determine the correct ending. Teney et al. [27] discuss the pitfalls of current out-of-distribution VQA methods. Perceptual score provide a way of identifying and quantifying these limitations.
Recently, Zadeh et al. [28] proposed SocialIQ, a dataset intended to reason about social situations in videos, specifically, emotion detection in a social situation. In this dataset, given a video and a question, the correct social situation should be recognized, e.g., “The man is upset because he is being insulted.” We show that it is easy to pick the correct statement using only the text data. This is possible because the statement’s correctness often correlates with the statement’s sentiment. Again, much like for VQA and AVSD, image information doesn’t seem to be necessary for reasonably accurate performance. Hence, biases exist which permit to ‘address the dataset’ without addressing the task. For SocialIQ, we attribute these biases to the fact that social reasoning is considered difficult, even for humans. Hence an annotator’s expertise is particularly important.
Importantly, going forward, we think it is elusive that we will be able to create unbiased datasets. We hence need techniques to automatically measure biases. In this work, we provide a mechanism that permits to do this for any dataset.
Methods to reduce bias: Several methods have been suggested to reduce bias from a dataset. Some techniques require prior knowledge of the biased variables, for instance, gender bias in vision is addressed by masking related features (e.g., faces) [29–38]. Also, some techniques require access to the test set to re-balance it [39]. Additionally, various approaches were proposed for visual question answering [40–43]. These methods use a classifier trained only on the question modality to regularize bias directly. Cadene et al. [40] suggest to mask a model’s softmax prediction with a softmax prediction of a subset classifier trained on the question modality. Clark et al. [41] use an ensemble method of a full classifier paired with a question subset classifier using products of experts [44]. Common to all those approaches is the use of a subset classifier to indicate reliance on subset of the data. In contrast, we assess the perceptiveness of the original model based on permutations of a subset of the data.
Feature importance methods: Also relevant to our work are techniques that measure featureimportance. These works generally focus on understanding why a model made a particular prediction. To this end, Ribeiro et al. [45] introduce LIME, a local explanation method that approximates a linear explanation model around a given example. Later, Shrikumar et al. [46] proposed DeepLift specifically for deep net models. Lundberg and Lee [47] introduced SHapley Additive exPlanations (SHAP), a method to estimate feature importance using an approximation of the features Shapley values. LIME assumes that the local model is linear, while SHAP does not have any such assumptions. Lundberg and Lee [47] show a connection between DeepLIFT and Shapely values introduced as ‘Deep SHAP.’ Each of those techniques concentrates on the importance of features, whereas we focus on the importance of a modality as a whole. Hessel et al. [48] proposed to assess how much a classifier benefits from interactions across different data modalities. To do this, EMAP approximates a multi-modal classifier with classifiers that each depend on only a single data modality. In contrast, we measure the perceptiveness of each modality separately and for every data point independently, i.e., we aim to identify how much a classifier relies on each data modality.

heading: 3 The Perceptual Score
text: The perceptual score quantifies the degree to which a model relies on the input data or a subset thereof. Said differently, the perceptual score assesses the importance of data or a subset of the data for a model’s result. For instance, if a model answers questions about an image without using cues extracted from the image modality, the model does not perceive the image modality. In this case we want the perceptual score for the image modality to be lower than that of a model which relies heavily on the image modality.
We believe that reporting the perceptual score of a model in addition to its accuracy is particularly important in a multi-modal setting. As a community we are developing increasingly complex models. However, to date we know very little about what parts of the data these models rely on. We think this lack in our understanding is due to a missing easy way to quantify how much a model relies on available data modalities. To rectify this we hope to illustrate that a simple yet intuitive score like the proposed perceptual score is very useful and easy to report too.
The following sections describe the perceptual score of a modality for a given model and dataset.

heading: 3.1 Setup
text: Let D = {(x1, y1), . . . , (x|D|, y|D|)} denote the test set data, where |D| refers to the number of samples in the dataset. Each sample is a pair consisting of the input data xi ∈ X and its corresponding label yi ∈ Y . Here, Y denotes a finite set of possible classes. In multi-modal datasets which we consider here, the data xi can be separated naturally into different parts as illustrated in Fig. 2a. For instance, in the SocialIQ task, we can partition the data into video-related, question-related and answer-related parts. Formally, letM = {M1, . . . ,M|M|} be a set of modalities of size |M|, e.g., the video-, the question- and the answer-modality. We partition the data xi into its modalities using a set notation, i.e., xi = {xM1i , . . . , x M|M| i }. We use x {M1,M2} i = {x M1 i , x M2 i } to refer to the first two modalities, i.e., the superscript can be a set.

heading: 3.2 Perceptual Score of a Data Modality
text: The perceptual score Pf,D(Mm) of a model f : X → Y towards modality Mm on data D is defined as
Pf,D(Mm) = 1
Z
( E(x,y)∼D[Pf,x,y(Mm)] ) , (1)
i.e., as the normalized expectation of sample perceptual scores Pf,x,y(Mm). Here, Z indicates the normalization factor, which can either be determined by the dataset alone (i.e., Z = ZD) or based on both the dataset and the model (i.e., Z = Zf,D). We discuss the normalization in Sec. 3.2.2.
The sample perceptual score Pf,x,y(Mm), outlined in Fig. 2b, aims to measure the degree to which a model f : X → Y relies on a modality Mm for prediction of sample x ∈ X . To do so we define the
sample perceptual score for modality Mm ∈M as the normalized difference between the accuracy of a model which uses all data modalities, and the accuracy of a model which doesn’t use modality Mm for prediction, i.e., as
Pf,x,y(Mm) = Accf,x,y(M)− Accf,x,y(M\ {Mm}). (2)
Here, M \ {Mm} is an operator that removes the influence of modality Mm from the set of all modalitiesM. We define this operation formally in Sec. 3.2.1. Note, Accf,x,y(M) refers to the classical prediction accuracy of a dataset sample (x, y) for a given trained model f withM the set of all modalities used for prediction, i.e., Accf,x,y(M) = 1f(xM)=y .
Intuitively, the sample perceptual score Pf,x,y(Mm) is high if the accuracy of a model that does not consider modality Mm for prediction is significantly smaller than the accuracy of a model which uses all data modalitiesM. Conversely, if the accuracy doesn’t change, irrespective of whether modality Mm is available or not, the model f doesn’t perceive the modality Mm. Note that in cases where the modality Mm irritates the model, the perceptual score can be negative.
In the following we discuss how to ‘remove’ a modality Mm from a model f (Sec. 3.2.1) and how to compute the normalization constant Z (Sec. 3.2.2).

heading: 3.2.1 Removing Modality Influence
text: Removing a modality from a trained model is difficult since typical models entangle modalities and compute high-order correlations. Ideally, we need a tool that minimizes the impact of one modality while maintaining the other components’ functionality. To achieve this, we study a permutation-based approach, i.e., we randomly permute the modality-related features among the test set data D. We think permutation is particularly useful for the perceptual score because it is hyper-parameter free. This ensures that the perceptual score defined in Eq. (1) is unambiguous.
Formally, to compute Accf,x,y(M\ {Mm}) we don’t use all modalities from data xi. Instead, we use all modalities but Mm, i.e., x M\Mm i and append the data x Mm j of modality Mm from another data point xj . Hereby j is drawn uniformly from {1, . . . , |D|}, i.e., from U(1, |D|). Taken together we compute the accuracy via
Accf,x,y(M\ {Mm}) = Ej∼U(1,|D|)[1f({xM\Mm ,xMmj })=y]. (3)
In the following we discuss how to compute the normalization Z employed in Eq. (1).

heading: 3.2.2 Normalization
text: Normalization enables comparability. For this, normalization aims for consistency of the perceptual score between models designed for the same task and between models designed for different tasks. Two types of normalization are useful to consider: 1) a task-normalization, which enables a more meaningful comparison of the perceptual score across different tasks; and 2) a model-normalization that enables a meaningful comparison of the perceptual score within the same task. We think models should be analyzed with both kinds of normalization in mind.
Task-normalization: Task normalization is particularly useful if a trivial classifier’s accuracy is high. Without normalization, the comparison of models designed for different tasks is inconsistent. For instance, if a task is relatively easy, the use of permuted data won’t modify the accuracy much because the task can be addressed with any data modality. This would result in a perceptual score close to zero, which isn’t compelling because, in this case, even a marginal reduction in performance might be significant.
The majority vote classifier always predicts the majority class of the employed training set. We use ÂccD to refer to the accuracy of the majority vote classifier evaluated on the test set D and compute the normalization factor via
ZD = 1− ÂccD. (4) Intuitively, the normalization factor is the gap between the perfect accuracy (i.e., 1) and the accuracy of the majority vote classifier, i.e., ÂccD. Hence, normalization by the majority vote accuracy
amplifies the score difference as we compare the obtained gain (classifier accuracy minus permuted classifier accuracy) to the maximally possible gain (one minus majority vote classifier accuracy).
We note that this normalization may result in perceptual score higher than one since the majority vote may be superior to the permuted accuracy. However, this case is unlikely in practice and did not occur in any of our experiments. Notably, this normalization is limited by the fact that the model accuracy is not considered, as we will discuss next.
Model-normalization: The performance of a model is an important factor. Suppose for two different classifiers permutation of a modality results in the same accuracy as that of the majority vote classifier. As a result, the initially stronger model will attain a higher perceptual score. This property can be desirable since one may want the perceptual score to reflect both perception and the ability to address a task. However, in this case the perceptual score does not solely reflect the degree to which a model considers a particular modality for decision making. Instead the perceptual score for a modality would be conflated with the model’s accuracy.
To obtain a score which solely reflects the degree to which a model considers a particular modality, we normalize by the model’s accuracy. Formally, we normalize via
ZD,f = E(x,y)∼D[Accf,x,y(M)]. (5)

heading: 4 Evaluation of Perceptual Scores
text: In the following, we assess the perceptual score using popular multi-modal datasets. Specifically, in Sec. 4.1, we study visual question answering (VQA, VQA-CP). We examine video social reasoning (i.e., SocialIQ) in Sec. 4.2. In Sec. 4.3, we assess visual dialog models. Our analysis shows that stateof-the-art models exploit biases that haven’t been documented. We study the bias by investigating samples with low perceptual scores and discover its cause.
Experimental setup: We compute the perceptual score based on five permutations per sample. We calculate the perceptual score five times with different permutations and report the mean score along with the standard deviations. We find the expectation to converge quickly and to be stable. For all the models, we used the official implementations. For more details please see our implementation.1

heading: 4.1 Visual Question Answering
text: The visual question answering task reasons about an image given a question. We use the VQAv2 dataset [20], which contains 443,757 image-question pairs in the train set and 214,354 in the validation set. We also assess the perceptiveness of models trained on ‘Visual Question Answering: Changing Priors’ (VQA-CP) data [21], which was released after several studies suggested that VQAv2 models heavily rely on answer priors. For instance, ‘how many’ questions are typically answered with ‘2.’ To overcome this shortcoming, VQA-CP suggested a new train-test-split. As a result, the train and test sets have different prior distributions for each question type. The new split consists of 438,183 training samples, and 219,928 samples for validation.
Baselines: We use four baselines for VQAv2: 1) BUTD [49], an early competitive approach that used detector-based features pre-trained on VisualGenome [50]; 2) BAN [51], a baseline that uses an effective multi-modal bilinear attention; 3) LMH [41], originally crafted for VQA-CP, this approach removes superficial question patterns; and 4) LXMERT [52] a large-scale Transformer-based model that is pre-trained with large amounts of image-text pairs. For VQA-CP, we also analyze: 1) CSS [53] which generates counterfactual training samples by masking critical objects in images or words in questions; 2) RMFE [42] which maximizes the amount of information that each modality contributes to the prediction.

heading: 4.1.1 Quantitative Analysis
text: In Tab. 1, we provide the perceptual score and the accuracy for different baselines and questions. We start by analyzing the perceptual scores of the vision and language modalities. In most cases, models perceive the question better (i.e., PQ > PV ). Studying the accuracy, LXMERT has the
1https://github.com/itaigat/perceptual-score
LXMERT Y/N 85.30 64.58 20.72 41.02 ± 0.40 24.29 ± 0.32 63.84 21.46 42.49 ± 0.32 25.16 ± 0.38 49.49 LMH Y/N 60.24 50.80 9.43 18.68 ± 0.33 15.66 ± 0.42 50.29 9.94 19.68 ± 0.29 16.50 ± 0.39 49.49 BAN Y/N 83.03 65.44 17.59 34.82 ± 0.36 21.18 ± 0.29 64.09 18.93 37.48 ± 0.33 22.80 ± 0.32 49.49 BUTD Y/N 80.94 65.42 15.52 30.73 ± 0.32 19.27 ± 0.33 64.24 16.69 33.05 ± 0.30 20.63 ± 0.25 49.49
VQA-CP
CSS All 57.89 36.46 21.43 26.43 ± 0.41 37.01 ± 0.27 7.93 49.96 61.61 ± 0.21 86.30 ± 0.31 18.91 RMFE All 54.20 29.91 24.29 27.11 ± 0.34 47.17 ± 0.31 7.65 46.55 51.95 ± 0.17 55.91 ± 0.24 10.40 LMH All 52.23 27.14 25.09 28.00 ± 0.32 45.35 ± 0.24 7.13 45.10 50.33 ± 0.14 54.08 ± 0.27 10.40
CSS Num 51.34 44.50 6.84 7.20 ± 0.38 13.32 ± 0.08 13.04 38.30 40.34 ± 0.29 74.60 ± 0.31 5.06 RMFE Num 44.03 37.25 6.77 7.13 ± 0.33 18.84 ± 0.33 34.05 9.97 10.51 ± 0.30 26.51 ± 0.33 5.06 LMH Num 37.40 30.35 7.05 7.43 ± 0.28 15.39 ± 0.28 27.48 9.92 10.45 ± 0.25 22.65 ± 0.28 5.06
CSS Other 46.48 10.12 36.36 37.57 ± 0.15 78.22 ± 0.15 4.12 42.36 43.77 ± 0.16 91.13 ± 0.14 3.23 RMFE Other 45.97 10.08 35.89 37.09 ± 0.17 77.94 ± 0.18 4.15 41.82 43.22 ± 0.11 91.14 ± 0.21 3.23 LMH Other 46.10 10.17 35.93 37.13 ± 0.18 78.07 ± 0.19 4.08 42.02 43.42 ± 0.11 90.96 ± 0.23 3.23
CSS Yes/No 83.11 82.52 0.59 0.71 ± 0.04 3.16 ± 0.04 43.84 39.27 100.0 ± 0.00 47.25 ± 0.33 64.46 RMFE Yes/No 74.47 62.49 11.98 18.58 ± 0.31 17.99 ± 0.31 59.31 15.16 23.51 ± 0.28 21.68 ± 0.31 35.52 LMH Yes/No 73.75 60.49 13.27 20.58 ± 0.32 16.08 ± 0.32 57.76 16.00 24.81 ± 0.29 20.35 ± 0.32 35.52
highest accuracy (68.97%). In contrast, LMH, which reduces the reliance on question priors, achieves the lowest accuracy (54.33%). However, the model-normalized perceptual score for the visual modality (PV /ZD,f ) suggests: LMH perceives image data similarly to other models. Studying the task normalized perceptual score for the visual modality (PV /ZD) suggests: despite the high accuracy on ‘Y/N’ questions, their image and question perceptual scores are very low, i.e., models mostly ignore the visual data and rely on priors.
Next, we show metrics for VQA-CP, a variant designed to reduce bias caused by answer priors. We compare different models on all the data by analyzing the model-normalized perceptual score for both visual and question modalities (PV /ZD,f and PQ/ZD,f ). Interestingly, the state-of-the-art model, CSS, has the lowest image perceptual score (37.01%) and the highest question perceptual score (86.30%), suggesting that the question modality may serve as a shortcut to answer without perceiving the image. Further, by analyzing the different question types via the task-normalized score (PV /ZD), we note that CSS has a perceptual score for the image modality of only 3.6% for ‘Y/N’ questions. One possible explanation: CSS generates new samples, which in turn alter priors perceived during training. We further investigate this hypothesis next.

heading: 4.1.2 Bias Analysis for CSS
text: The Counterfactual Samples Synthesizing (CSS) model produces counterfactual training samples by masking either critical objects in images or words in questions, and by assigning different groundtruth answers. Our perceptiveness study above shows that the CSS model has a significantly lower perceptual score for the visual modality, despite being state-of-the-art on VQA-CP with a substantial accuracy gap of 6.5% over LMH. Why?
The first point to note: CSS generates new samples which may shift prior distributions leveraged during training. Recalling that VQA-CP was introduced to prevent benefiting from answer priors in VQAv2 reveals a potential reason for the improvements of CSS.
Use of the sample perceptual score permits a more in-depth analysis. In Tab. 2, we identify popular question start tokens with low sample perceptual scores for the visual modality (‘do,’ ‘has,’ ‘can,’ ‘is a’). We further examine their prediction accuracy using both CSS and LMH. We find that the proportion of ‘yes’ and ‘no’ answers between the train and the test set differ: the yes answer is correct for 88% of the questions in the test set starting with ‘has,’ while the yes answer is never correct for the corresponding training set questions. For CSS, counterfactual samples, however, produce equal proportions. As a result, CSS seemingly alleviates the prior inconsistency and adjusts the majority of its predictions to ‘yes,’ which more closely resembles the test set.
Use of the perceptual scores hence permits to hypothesize: improvements in CSS can be attributed to a shifted prior distribution instead of a complex counterfactual data-manipulation. To test this hypothesis we train the LMH model using samples obtained from CSS training. Importantly, we did not modify the image or the question. Without even any changes to the input, we obtain an accuracy of 57.54%, only 0.3% lower than CSS and within standard deviation.

heading: 4.2 Video Social Reasoning
text: SocialIQ [28] proposes an unconstrained benchmark, specifically designed to understand social situations. More concretely, given an input tuple of a video, a question, and an answer, the task is to predict whether the answer is correct or not. The videos were collected from YouTube and annotated by students. The dataset is split into 37,191 training samples, and 5,320 validation set samples.
Baseline: We were able to reproduce the SocialIQ baseline [28] and achieve an accuracy of 64.84% using all feature modalities during training. In fact, we achieved an accuracy of 67.38% by changing the baseline’s model to FGA [54] and by using GloVe [55] instead of BERT features [56].

heading: 4.2.1 Quantitative Analysis
text: In Tab. 3 we show scores for different metrics. The task-normalized perceptual scores (PM/ZD) for different modalities M (first column) of FGA and the baseline reveals heavy reliance on the answer modality and low dependence on the video modality (i.e., 15.24 vs. 3.25). To verify that the answer is indeed the only important modality, we train a classifier using only the answer modality and observe: we can surpass the original paper’s baseline by 4% when only using answer features, achieving 68.65% accuracy. Again, the perceptual score helped us understand shortcomings of a model. Upon analyzing the bias source, we observe a sentiment bias, which we discuss next.

heading: 4.2.2 Bias Analysis for SocialIQ
text: Assessing samples with low sample perceptual scores can reveal biases. Our study suggests that the labels correlate strongly with the sentiment. In Fig. 3, we marked with a red border two videos with a low sample perceptual score for the video modality. Studying those and similar samples, we find:
1) when the answer is True, the answer has a positive sentiment; 2) in contrast, when the answer is False, the answer contains words with a negative connotation, e.g., ‘uncomfortable.’
We hypothesize: successful prediction of the answer by just looking at the answer modality for SocialIQ data is due to sentiment-biased annotations. To validate this hypothesis, we use an offthe-shelf sentiment classifier from the NLTK package [57]. When applied to SocialIQ without any training, we obtain a remarkable answer prediction accuracy of 66.7%. This matches our reported result of 68.65% quite reasonably and outperforms the SocialIQ baseline [28].

heading: 4.3 Visual Dialog
text: The visual dialog task encourages models to ask and answer questions about visual input. Notably, each dialog-interaction employs many modalities (e.g., image, question, caption, dialog-history). We show our results on the VisDial v1.0 dataset, where 123,287 images are used for training, 2,000 images for validation, and 8,000 images for testing [58]. Each image is associated with ten questions, and each question has 100 corresponding answer candidates.
Instead of accuracy Accf,x,y(M), here, we use ranking-based metrics as the Visual Dialog dataset primarily uses two metrics: MRR and NDCG. In short, the MRR metric examines the rank of a single ground-truth response (i.e., sparse annotations), while the NDCG metric measures the cumulative gain in the case of multiple correct answers (i.e., dense annotations). We computed the majority ranking based on answer frequency in the train set. See appendix for more.
Baselines: We use two baselines for VisDial v1.0: 1) FGA [54], an attention unit inspired from graphical models that infers an attention map for each modality; and 2) LS [59], which pre-trains on related vision-language datasets, e.g., Conceptual Captions and Visual Question Answering [1, 60]. We also report LS (CE), which finetunes on the dense annotations, at the expense of MRR performance.

heading: 4.3.1 Quantitative Analysis
text: In Tab. 4, we show the perceptual scores for different modalities (i.e., M ), and for two metrics: MRR and NDCG. We find: 1) using the MRR metric, the model-normalized perceptual score for the question modality of FGA is relatively high compared to LS (i.e., 51.26% vs. 30.49%). However, when we employ the NDCG metric, the LS model perceives the question better (i.e., 63.14% vs. 45.11%). 2) analyzing the only model optimized for NDCG, i.e., LS (CE), reveals: When analyzed using the NDCG metric, the model-normalized perceptual score for both image and question is significantly higher than for MRR. Interestingly, the LS (CE) model-normalized perceptual score for the image modality is on par with the other models when we use the MRR metric. However, the question perceptual score is low, which suggests the reason for the low MRR performance may be related to the low utilization of the question. Finally, we note that all models, on both metrics, seem to ignore the caption, which suggests caption information is redundant.

heading: 4.4 Audio-visual Crowd Counting
text: We further investigate a model for audio-visual crowd counting task. Given an image and an utterance, the task is to count the number of people in a scene. We use the auDiovISual Crowd cOunting (DISCO) dataset, consisting of 1,935 images and the corresponding audio clips, and 170,270 annotated instances. Hu et al. [61] hypothesize that in extreme cases (i.e., low-quality images), the scene audio can improve the prediction. As expected, the perceptual score is able to quantify the degree to which each modality is perceived. In Tab. 5 we compare the low-quality and high-quality images by employing task normalization. It reveals that in high-quality image settings, the audio modality is mostly ignored (the task-normalized audio perceptual score is 9.22%). In contrast, the audio modality perceptiveness for low-quality image settings is significant (the task-normalized audio perceptual score is 23.91%).

heading: 5 Conclusion
text: We introduced the perceptual score of a multi-modal classifier towards a data modality. The perceptual score assesses a classifier’s perceptiveness of a modality and reveals exciting insights if analyzed carefully. We hope that this is demonstrated by our studies which reveal that 1) shifted prior distributions seem to help CSS achieve state-of-the-art results, 2) SocialIQ data exhibits a sentiment bias, 3) Visual Dialog caption information appears less informative, and 4) Audio perceptiveness for audio-visual crowd counting increases when low-quality images were present. We hope that researchers working on multi-modal models see the use of the perceptual score and will start to report the perceptual score in addition to classical accuracy metrics.
Limitations: We propose perceptual scores, a novel metric that conveys information about multimodal classifiers. The two limitations we can see: 1) a small computational overhead; 2) a false sense of security. Perceptual scores don’t alleviate the need to carefully study results. However we think the societal and scientific benefits of reporting this novel metric outweigh the concerns.
Acknowledgements: This work is supported in part by NSF Grant #1718221, 2008387, 2045586, 2106825, MRI #1725729, and NIFA 2020-67021-32799. We also acknowledge support from Samsung, Amazon, and Cisco Systems Inc. (Gift Award CG 1377144/thanks for access to Arcetri).

Review summaries:
The paper proposes a score to understand biases in datasets and models by trying to assess the reliance of a model on different modalities. Specifically, this perceptual score is defined as the difference between the model’s accuracy with all modalities and with one modality removed. The proposed method is  used to bring insights into significance of text and visual modalities for different question answering systems. 

This paper proposes a measure for evaluating multimodal models on how much they take into account the various modalities in making their predictions. The proposed "perceptual score" evaluates a model's performance on each example when one of its modalities is randomly sampled from other items in the dataset (an expectation of performance when paired with all possible versions of the modality being tested). The general idea is that if the performance when paired with a random sample is similar to the performance when paired with the ground truth modality, then the model is likely not taking into account that modality when making its prediction.  The paper evaluates the metric on several vision and language tasks (VQA, VQA-CP, SocialIQ, and VisDial) with several interesting findings (e.g., that one can achieve relatively high performance on SocialIQ without even training a model and only considering one of the modalities).

This paper pays attention to analyzing the perceptiveness of multi-modal classifiers, proposing a perceptual score, which is very helpful for the community. Using the proposed perceptual score, the authors analyze a wide range of models on different datasets, finding that multi-modal models tend to ignore some modalities while taking shortcuts. In terms of bias, this paper finds that SocialIQ is biased by sentiment. The above interesting findings are helpful for researchers to propose new models and develop new datasets.