Perceptual Score: What Data Modalities Does Your Model Perceive?

1 Introduction . Machine learning advances over the last decade are remarkable . Challenges that seemed daunting merely ten years ago are now a breeze , and new applications that we barely dared to dream about seem achievable within the next few years . Indeed , accuracy metrics on tasks like visual question answering and reasoning suggest significant improvements . Reported improvements are to a large extent due to the availability of large datasets [ 1–3 ] , computational performance advances , e.g. , for GPUs , and a better understanding about how to encode inductive biases into deep-nets , e.g. , by using rectified linear units [ 4 ] , normalization [ 5 ] , skip connections [ 6 ] , transformers [ 7 ] , etc . However , importantly , developed deep-net architectures are not guaranteed to solve a given task . There is a chance that they may instead exploit dataset biases . This concern is surely in part due to non-robust training techniques , and a plethora of methods improve classifier robustness [ 8–10 ] . However , datasets play an important role in controlling the extracted bias as well . For instance , if correct answers in a question-answering task are significantly shorter than incorrect ones , classifier training should not use answer length as a cue . Although this seems reasonable , for audio-visual scene aware dialog , Schwartz et al . [ 11 ] find for example that in many cases the question alone is sufficient to generate a scene-aware dialog response , avoiding the need to look at the video . Hence , in order to assess the suitability of a classifier , we need to understand how much it relies on different data modalities . To quantify how much a classifier relies on its different input modalities , we introduce the perceptual score . The perceptual score assesses the degree to which a model relies on a modality . To do so the perceptual score permutes the features of a modality across samples in the test set after the classifier 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . How many animals are pictured ? What color is the car ? Why are the men jumping ? What color is the ground ? Im ag e Pe rc ep tu al S co re Q ue st io n Pe rc ep tu al S co re Permute Image Permute Question How many animals are pictured ? Original Data How many animals are pictured ? Permuted Test ( a ) Visual question answering ( VQA ) data . Modality perceptiveness is measured through a permutation test . LMH ( 2019 ) RMFE ( 2020 ) CSS ( 2020 ) 40 50 60 70 80 Accuracy Image Perceptual Score Question Perceptual Score ( b ) Progress of VQA-CP models . Figure 1 : Multi-modal datasets often have undesired biases : ( a ) To identify those biases we suggest the perceptual score as a new metric . It assesses the change in prediction when a model ’ s input for some modalities is permuted during testing . If the classifier output remains identical despite permutation , a model doesn ’ t perceive the modality . ( b ) Using the perceptual score we identify that recent progress of VQA models may not be entirely due to better reasoning . was trained , as illustrated in Fig . 1a . If the classifier ’ s performance drops to or below chance level , the perceptual score is high . This intuitively applies to single-modality models too : randomly permuting test data and labels after training results in chance-level classification accuracy . Using the perceptual score , we find a surprisingly consistent trend across five popular datasets ( VQA , VQA-CP , VisDial , SocialIQ , DISCO ) : recent , more accurate state-of-the-art multi-modal models for visual question-answering or visual dialog tend to perceive the visual data less than their predecessors ( see Fig . 1b ) . This trend is concerning as answers are hence increasingly inferred from textual cues only . Using the perceptual score also helps to analyze model biases by decomposing the score into data subset contributions . For example , the perception of an image and question varies depending on the question type . None of the recent VQA-CP models showed high image perception scores for ‘ number ’ -type questions . A surprisingly low image perception score is obtained for the state-of-the-art model when confronted with ‘ yes/no ’ questions . We hope the perceptual score spurs a discussion regarding the perceptiveness of multi-modal models and we also hope to encourage the community working on multi-modal classifiers to start quantifying perceptiveness of models . Our contributions : • We propose the perceptual score , a simple yet effective method for assessing the perceptiveness of multi-modal models towards a modality . • Our experiments span multiple datasets and models . We find that multi-modal models tend to ignore some modalities while taking shortcuts . • Subsequently , we investigate the sources of bias on popular multi-modal datasets such as VQA-CP and SocialIQ : We find that SocialIQ is biased by sentiment , and bias in the VQA-CP model results from shifting the training priors to more closely resemble those in the test set . 2 Related Work . The perceptual score assesses the degree to which a classifier relies on a particular input modality . This is related to studying datasets and their biases , methods which aim to reduce the biases captured by classifiers and work which studies the importance of features . We review all three areas next . Datasets and bias : Data has been a central element for machine learning progress [ 12–14 ] in the last two to three decades . The ImageNet challenge [ 2 ] and the development of AlexNet [ 15 ] sparked the deep learning era . But as datasets grow , biases emerge which may go undetected for a long time . For instance , the background in ImageNet can reveal information about the object class [ 16 ] . Also , with the increasing popularity of crowdsourcing systems like Amazon Mechanical Turk , many datasets are annotated in uncontrolled environments . Different annotators are hence injecting unknown socio-economic properties into dataset annotations [ 17 ] . Those dataset biases can be detrimental to the considered task . For instance , meticulously collected visual question answering ( VQA ) data [ 1 ] aims to provide a platform for exciting research to advance image-language understanding . However , it is non-trivial to remove biases from this type of data . Indeed , it was reported that the question solely is sufficient to detect the correct answer [ 18 , 19 ] , i.e. , no image information is required . In an attempt to fix this bias , the dataset was re-annotated [ 20 ] , or the train and test split were reorganized [ 21 ] . Similarly , Schwartz et al . [ 11 ] reported that in Audio-visual-scene-aware dialog ( AVSD ) [ 22 ] , the question cue is often stronger , making the desired video and sound reasoning implausible or unnecessary . Likewise , SNLI [ 23 ] , aims to determine the correctness of a hypothesis given a premise . However , Gururangan et al . [ 24 ] point out that an internal bias exists : linguistic features not related to the premise correlate with the label . Further , the Story Cloze [ 25 ] dataset permits to develop models which estimate the correct ending of a story . Schwartz et al . [ 26 ] show that , for this dataset , length alone is a powerful feature to determine the correct ending . Teney et al . [ 27 ] discuss the pitfalls of current out-of-distribution VQA methods . Perceptual score provide a way of identifying and quantifying these limitations . Recently , Zadeh et al . [ 28 ] proposed SocialIQ , a dataset intended to reason about social situations in videos , specifically , emotion detection in a social situation . In this dataset , given a video and a question , the correct social situation should be recognized , e.g. , “ The man is upset because he is being insulted. ” We show that it is easy to pick the correct statement using only the text data . This is possible because the statement ’ s correctness often correlates with the statement ’ s sentiment . Again , much like for VQA and AVSD , image information doesn ’ t seem to be necessary for reasonably accurate performance . Hence , biases exist which permit to ‘ address the dataset ’ without addressing the task . For SocialIQ , we attribute these biases to the fact that social reasoning is considered difficult , even for humans . Hence an annotator ’ s expertise is particularly important . Importantly , going forward , we think it is elusive that we will be able to create unbiased datasets . We hence need techniques to automatically measure biases . In this work , we provide a mechanism that permits to do this for any dataset . Methods to reduce bias : Several methods have been suggested to reduce bias from a dataset . Some techniques require prior knowledge of the biased variables , for instance , gender bias in vision is addressed by masking related features ( e.g. , faces ) [ 29–38 ] . Also , some techniques require access to the test set to re-balance it [ 39 ] . Additionally , various approaches were proposed for visual question answering [ 40–43 ] . These methods use a classifier trained only on the question modality to regularize bias directly . Cadene et al . [ 40 ] suggest to mask a model ’ s softmax prediction with a softmax prediction of a subset classifier trained on the question modality . Clark et al . [ 41 ] use an ensemble method of a full classifier paired with a question subset classifier using products of experts [ 44 ] . Common to all those approaches is the use of a subset classifier to indicate reliance on subset of the data . In contrast , we assess the perceptiveness of the original model based on permutations of a subset of the data . Feature importance methods : Also relevant to our work are techniques that measure featureimportance . These works generally focus on understanding why a model made a particular prediction . To this end , Ribeiro et al . [ 45 ] introduce LIME , a local explanation method that approximates a linear explanation model around a given example . Later , Shrikumar et al . [ 46 ] proposed DeepLift specifically for deep net models . Lundberg and Lee [ 47 ] introduced SHapley Additive exPlanations ( SHAP ) , a method to estimate feature importance using an approximation of the features Shapley values . LIME assumes that the local model is linear , while SHAP does not have any such assumptions . Lundberg and Lee [ 47 ] show a connection between DeepLIFT and Shapely values introduced as ‘ Deep SHAP. ’ Each of those techniques concentrates on the importance of features , whereas we focus on the importance of a modality as a whole . Hessel et al . [ 48 ] proposed to assess how much a classifier benefits from interactions across different data modalities . To do this , EMAP approximates a multi-modal classifier with classifiers that each depend on only a single data modality . In contrast , we measure the perceptiveness of each modality separately and for every data point independently , i.e. , we aim to identify how much a classifier relies on each data modality .

Summary: The paper proposes a score to understand biases in datasets and models by trying to assess the reliance of a model on different modalities. Specifically, this perceptual score is defined as the difference between the model’s accuracy with all modalities and with one modality removed. The proposed method is  used to bring insights into significance of text and visual modalities for different question answering systems.