MultiViz: Towards Visualizing and Understanding Multimodal Models
Authors: Paul Pu Liang
Yiwei Lyu
Gunjan Chhablani
Nihal Jain
Zihao Deng
Xingbo Wang
Louis-Philippe Morency
Ruslan Salakhutdinov
Venue: ICLR 2023 poster (2023)
Keywords: multimodal learning
representation learning
interpretation
visualization

Abstract: The promise of multimodal models for real-world applications has inspired research in visualizing and understanding their internal mechanics with the end goal of empowering stakeholders to visualize model behavior, perform model debugging, and promote trust in machine learning models. However, modern multimodal models are typically black-box neural networks, which makes it challenging to understand their internal mechanics. How can we visualize the internal modeling of multimodal interactions in these models? Our paper aims to fill this gap by proposing MultiViz, a method for analyzing the behavior of multimodal models by scaffolding the problem of interpretability into 4 stages: (1) unimodal importance: how each modality contributes towards downstream modeling and prediction, (2) cross-modal interactions: how different modalities relate with each other, (3) multimodal representations: how unimodal and cross-modal interactions are represented in decision-level features, and (4) multimodal prediction: how decision-level features are composed to make a prediction. MultiViz is designed to operate on diverse modalities, models, tasks, and research areas. Through experiments on 8 trained models across 6 real-world tasks, we show that the complementary stages in MultiViz together enable users to (1) simulate model predictions, (2) assign interpretable concepts to features, (3) perform error analysis on model misclassifications, and (4) use insights from error analysis to debug models. MultiViz is publicly available, will be regularly updated with new interpretation tools and metrics, and welcomes inputs from the community.

Published as a conference paper at ICLR 2023
MULTIVIZ: TOWARDS VISUALIZING AND UNDERSTANDING MULTIMODAL MODELS
Paul Pu Liang', Yiwei Lyu², Gunjan Chhablani", Nihal Jain', Zihao Deng', Xingbo Wang', Louis-PPilippe Moreney', Ruslan Salakhutdinov!
'Carnegie Mellon University, UUniversity of Michigan, .Georgia Tech, 4HKUST https: //t.//g/ii /gthtbb.t.hhtt.gggg com/
ABSTRACT
The promise of multimodal models for real-world applications has inspired re- search in visualizing and understanding their internal mechanics with the end goal of empowering stakeholders to visualize model behavior, perform model debug- ging. and promote trust in machine learning models. However, modern multimodal models typically black-box neural networks, which makes it challenging understand their internal mechanics. How can we visualize the internal modeling multimodal interactions in these models? Our paper aims fill this gap by propos- ing MULTI ViZ, a method for analyzing the behavior of multimodal models by scaffolding the problem of interpretability into 4 stages: (1) unimodal importance: how each modality contributes towards downstream modeling and prediction, (2) cross-modal interactions: how different modalities relate with each other, (3) multi- modal representations: how unimodal and cross-modal interactions are represented in decision- features, and (4) multimodal prediction: how decision-level fea- tures are composed to make a prediction. MULTIVIZ is designed to operate on diverse modalities, models, tasks, and research areas. Through experiments on 8 trained models across 6 real-world tasks, show that the complementary stages in MULTIVIZ together enable users to (1) simulate model predictions, (2) assign interpretable concepts features, (3) perform error analysis on model misclas- sifications, and (4) use insights from error analysis to debug models. MUL.TIVIZ is publicly available, will be regularly updated with new interpretation tools and metrics, and welcomes inputs from the community.
INTRODUCTION
The recent promise of multimodal models that integrate information from heterogeneous sources of data has led their proliferation in numerous real-world settings such as multimedia (Naphade et al., 2006), affective computing (Poria et al., 2017), robotics (Lee et al., 2019), and healtheare (Xu al., 2019). Subsequently, their impact towards real-world applications has inspired recent research in visualizing and understanding their internal mechanics (Liang et al., 2022; Goyal et al., 2;16; Park et al., 2018) as step towards accurately benchmarking their limitations for more reliable deployment (Hendricks et al., 2018; Jabri et al., 2016). However, modern parameterizations of multimodal models are typically black-box neural networks, such as pretrained transformers (Li et al., 2019; Lu et al., 2019). How can we visualize and understand the internal modeling of multimodal information and interactions in these models?
As step in interpreting multimodal models, this paper introduces analysis and visualization method called MULTIVIZ (see Figure 1). To tackle the challenges of visualizing model behavior, we scaffold the problem of interpretability into stages: (1) unimodal importance: identifying the contributions of each modality towards downstream modeling and prediction, (2) cross-modal interactions: uncovering the various ways in which different modalities can relate with each other and the types of new information possibly discovered as a result of these relationships, (3) multimodal representations: how unimodal and cross- modal interactions are represented in decision- features, and (4) multimodal prediction: how decision-level features are composed to make a prediction for given task. In addition to including current approaches for unimodal importance (Goyal et al., 2016; Merrick and Taly, 2020; Ribeiro et al., 2016) and cross-modal interactions (Hessel and Lee, 2020; Lyu et al., 2022), we additionally propose new methods for interpreting cross- moosl interactions, multimodal representations, and prediction to complete these stages in MULTIVIZ. By viewing multimodal interpretability through the lens of these 4 stages, MULTIVIZ contributes a modular and human-in-the--lop visualization toolkit for the community to visualize popular multimodal

---

Published as a conference paper at ICLR 2023
Local analysis of datapoint
Multimodal
Unimodal importance What color the building?
Multimodal representations
Multimodal prediction
representations
"color"
2. Cross-modal interactions
"color"
+0.8
is building?
Red
"building"
-0.4
"people" 0.3
is building?
analysis by retrieving similar datapoints
Satsbury Rd sign? checkers on the wonh?
Figure 1: Left:
We scaffold the problem of multimodal interpretability and propose MULTIVIZ.
comprehensive analysis method encompassing a set of fine-grained analysis stages: (1) unimodal importance identifies the contributions of each modality, (2) cross-modal interactions uncover how different modalities relate with each other and the types of new information possibly discovered as result of these relationships, (3) multimodal representations study how unimodal and cross modal interactions are represented in decision-leve. features, and (4) multimodal prediction studies how these features are composed to make prediction. Right: We visualize multimodal representations through local and global analysis. Given an input datapoint, local analysis visualizes the unimodal and cross-modal interactions that activate a feature. Global analysis informs the user of similar datapoints that also maximally activate that feature, and is useful in assigning human-interpretabby concepts to features by looking at similarly activated input regions (e.g., the concept of color).
datasets and models as well as compare with other interpretation perspectives, and for stakeholders to understand multimodal models in their research domains.
MULTIVIZ is designed to support many modality inputs while also operating on diverse modalities, models, tasks, and research areas. Through experiments 6 real- world multimodal tasks (spanning fusion, retrieval, and question- antweoinnn,,rrng)..
modalities, and 8 models, we show that MULTIVIZ helps gain deeper understanding of model behavior measured via proxy task of model simulation. We further demonstrate that MULTIVIZ helps human users assign interpretable language concepts previously uninterpretable features and perform error analysis on model misclassification. Finally, using takeaways from error analysis, we present case study of human-in- in-the--ohoopp model debugging. Overall, MULTIVIZ provides practical toolkit for interpreting multimodal models for human understanding and debugging. MULTIVIZ datasets, models, and code are at ht https: / //gittttte srgithub.r. com/plmm/o..o. a 79 MultiViz.
2 MULTIVIZ: VISUALIZING AND UNDERSTANDING MULTIMODAL MODELS
This section presents MULTIVIZ, our proposed analysis framework for analyzing the behav- ior of multimodal models. As a general setup, we assume multimodal datasets take the form D = (x1, X2, 3) = {(x (), ..... 2/2 22 ..... ") with boldface x denoting the entire modality, each æ1, 22 indicating modality atoms (i.e., fine-grained sub-parts of modalities that would like to analyze, such as individual words in a sentence, object regions in an image, or time-steps in time-series data), and y denoting the label. These datasets enable us train multimodal model y = f(xi, X2; 0) which we are interested in visualizing.
Modern parameterizations of multimodal models f are typically black-box neural networks, such as multimodal transformers (Hendricks et al., 2021; Tsai et al., 2019) and pretrained models (Li et al., 2019; Lu al., 2019). How can we visualize and understand the internal modeling of multimodal information and interactions in these models? Having an accurate understanding of their decision- making process would enable us benchmark their opportunities and limitations for more reliable real-world deployment. However, interpreting ʃ is difficult. In many multimodal problems, it is useful to first scaffold the problem of interpreting ff into several intermediate stages from low-level unimodal inputs to high-level predictions, spanning unimodal importance, cross-modal interactions, multimodal representations, and multimodal prediction. Each of these stages provides complementary information the decision-making process (see Figure 1). We now describe each step in detail and propose methods to analyze each step.
2.1 UNIMODAL IMPORTANCE (U)
Unimodal importance aims to understand the contributions of each modality towards modeling and prediction. It builds upon ideas of gradients (Simonyan al., 2013; Baehrens et al., 2010; Erhan
2

---

Published as a conference paper at ICLR 2023
CLEVR
VQA
MM-IMDb
Flickr-30k
CMU-MOSEI
MARS
The other small shiny thing thot How many birds? the some shape as the tiny yellow shiny object
he can breathe the Mortion air...
Three small dogs, two white and one black and white, sidewolk.
Why am spending my money watching this? (sigh) think was more sad...
color?
Figure 2: Examples of cross-modal interactions discovered by our proposed second-order gradient approach: first taking gradient of model f with respect to an input word (e.g., g., 21 birds), before taking a second-ordes gradient with respect to all image pixels (highlighted in green) or bounding boxes (in red boxes) X2 indeed results in all birds in the image being highlighted.
al., 2009) and feature attributions (e.g., LIME (Ribeiro et al., 2016), Shapley values (Merrick and Taly, 2020)). We implement unimodal feature attribution methods as module UNT(Je, y. x) taking in a trained model fe. an output/feature y which analysis is performed with respect to, and the modality of interest x. UNI returns importance weights across atoms of modality
2.2 CROSS CROSS-MODAL INTERACTIONS (C)
Cross-modal interactions describe various ways in which atoms from different modalities can relate with each other and the types of new information possibly discovered result of these relationships. Recent work (Hessel and Lee, 2020; Lyu et al., 2022) has formalized a definition of cross-modal interactions by building upon literature in statistical non-additive interactions:
Definition 1 (Statistical Non-Additive Interaction (Friedman and Popescu, 2008; Sorokina al., 2008; Tsang et al., 2018 8; 2019)). A function f learns a feature interaction I between 2 unimodal atoms 2 and 22 if and only if f cannot be decomposed into sum of unimodal subfunctions 91, 92 such that f(z1,, 22) = 9i (x1 + (2(2)).
This definition of non-additive interactions is general enough to include different ways that interactions happen. including multiplicative interactions from complementary views of the data (i.e., an interaction term x Waz (Jayakumar et al., 2020)), or cooperative interactions from equivalent views (i.e., interaction term majority (=))) [ [x²)) (Ding and Tibshirani, 2021)). Using this definition, MIZ first includes two recently proposed methods for understanding cross-modal interactions: EMAP (Hessel and Lec, 2020) decomposes f(x1, = 91 (z1) + 92 (22)
(1) into strictly unimodal representations 91, 92, and cross-modal representation U12 f E.,, (f) Ex, (f) Eu, (f)) to quantify the degree of global cross- modal interactions across an entire dataset. DIME (Lyu et al., 2022) further extends EMAP using feature visualization on each disentangled representation locally (per datapoint). However, these approaches require approximating expectations over modality subsets, which may not scale beyond modalities. To fill this gap, we propose an efficient approach for visualizing these cross-modal interactions by observing that the following gradient definition directly follows from Definition 1:
Definition 2 (Gradient definition of statistical non-additive interaction). A function ʃ exhibits
0 [(x)
non-additive interactions among unimodal atoms 2 and z2 if Ex B22
02,
Taking a second-order gradient of f zeros out the unimodal terms gi (x1) and (2(22) and isolates the interaction (12(21, Theoretically, second-order gradients are necessary and sufficient to recover cross-modke interactions: purely additive models will have strictly second-ortes gradients so 8 (x) 0, and any non-linear interaction term 912(11, x22 has non-zero second-order
E .23
8² f(m)
gradients since cannot be a constant or unimodal function, so Ex
> 0.
Definition 2 inspires us to extend first-order gradient and perturbation- based approaches (Han et al., 2020; Ribeiro et al., 2016; Yosinski et al., 2015) to the second order. Our implementation first computes a gradient of f with respect to a modality atom which the user is interested in querying cross-modal interactions for (e.g., σ = birds), which results in a vector V, 8f of the same dimension as 21 (i.e., token embedding dimension). We aggregate the vector components of V, via summation produce single scalar ||V before taking a second-order gradient with respect
Te,
3

---

Published as a conference paper at ICLR 2023
to all atoms of the second modality 22 X2 (e.g., all image pixels), which results in vector of the same dimension as X2 (i.e., total number of pixels). Each 0210xa (1) 8x, Dag
12 0'1
scalar entry in 112 highlights atoms x2 that have non-linear interactions with the original atom ø1. and we choose the 22 's with the largest magnitude of interactions with 21 (i.e., which highlights the birds in the image, see Figure 2 for examples on real datasets). We implement a general module CM(Ja, y; 21, X2) for cross-modal visualizations, taking in trained model fo, an output/feature y, the first modality's 's atom of interest 2 and the entire second modality of interest X2, before returning importance weights across atoms 22 of modality X2 (see details in Appendix A.2).
2.3 MULTIMOOAL REPRESENTATIONS
Given these highlighted unimodal and mrosl interactions at the input level, the next stage aims to understand how these interactions are represented at the feature representation level. Specifically, given trained multimodal model f. define the matrix M, R Nxd as the penultimate layer of representing (uninterpretablle deep feature representations implicitly containing information from both unimodal and cross-modal interactions. For the ith datapoint, = Ma(1) collects set of individual feature representations 21, 22,..., Zd E R. We aim to interpret these feature representations through both local and global analysis (see Figure (right) for an example):
Local representation analysis (Re) informs the user on parts of the original datapoint that activate feature 2j. To do so, we run unimodal and cross-modal visualization methods with respect feature 2j (i.e., UNI( (fv 2z, x). CM(fo. 2j. æ X2)) in order to explain the input unimodal and cross-modal interactions represented in feature 2j. Local analysis is useful in explaining model predictions on the original datapoint by studying the input regions activating feature 2j.
Global representation analysis (Rg) provides the user with the top it datapoints Dx(2))
{(x), X2, ) that also maximally activate feature 2j. By further unimodal and cross-modal visualizations on datapoints in (24),, global analysis is especially useful in helping humans assign interpretable language concepts to each feature by looking at similarly activated input regions across datapoints (e.g., the concept of color in Figure 1, right). Global analysis can also help to find related
datapoints the model also struggles with for error analysis.
2.4 MULTIMODAL PREDICTION (P)
Finally, the prediction step takes the set of feature representations 21, 22, ..., 2d and composes them to form higher-level abstract concepts suitable for a task. We approximate the prediction process with linear combination of penultimate layer features by integrating a sparse linear prediction model with neural network features (Wong et al., 2021). Given the penultimate layer M, R we fit linear model (Y X = x) = M_ 3 (bias 30 omitted for simplicity) and solve for sparsity using:
1 2N
i arg min
M B v1 1111 + 22118112
8
The resulting understanding starts from the set of learned weights with the highest non-zero coef- ficients Seop = {(())),(()).... and corresponding ranked features zoo = {=(1)) =(2), ...]. Prop tells the user how features Sop composed to make prediction, and zop can then be visualized with respect to unimodal and cross- modal interactions using the representation stage (Section 2.3).
2.5 PUTTING EVERYTHING TOGETHER
We summarize these proposed approaches for understanding each step of the multimodal process and show the overall MULTIVI: IZ user interface in Figure 3. This interactive API enables users choose multimodal datasets and models and be presented with a set of visualizations at each stage, with an overview page for general unimodal importance, cross-modal interactions, and prediction weights, as well as feature page for local and global analysis of user-selected features (see Appendix B for more algorithm and user interface details).
EXPERIMENTS
Our experiments are designed to verify the usefulness and complementarity of the 4 MULTIVIZ stages. We start with model simulation experiment to test the utility of each stage towards overall model understanding (Section 3.... We then dive deeper into the individual stages by testing how well MULTIVIZ enables representation interpretation (Section 3.2) and error analysis (Section 3.3), before presenting a case study of model debugging from error analysis insights (Section 3.4). We showcase the following selected experiments and defer results on other datasets to Appendix D.
4

---

Published as a conference paper at ICLR 2023
MuhiViz Website
Overview Page
Cueedes of cles 23531g analyds
Features Page Festure analyds for dhex 21531gian
Local anahpiis for original datapoint
U/ UkiLfo.5.x) Unimmdde image
C - CML S. Xil Crovemodal beat-o Image interaction
fopazs 0 Muttmodal fostures and prodiction
Importance
Gobal analysk across deiltr datapolntt Jude
Unilfe Sot)
OML &. Xe) Mel Interaction
Unimodaa
Cross
Importance
ylu 2412
Figure 3: MULTIVIZ provides interactive visualization API across multimodal datasets and models. The overview page shows general unimodal importance, cross-modal interactions, and prediction weights, while the features page enables local and global analysis of specific user-selected features.
Table I: MULTIVIZ enables fine-grained analysis across 6 datasets spanning 3 research areas, 6 input modalities (f: language, i: image, v: video, a: audio, time-series, la: tabular), and models.
Area
Dataset
Model
Modalities
Samples
Prediction task sentiment, emotions movic genre classification
CMU-MOSEL MM-IMDB
MULT
a, +
22, 777
Fusion
LRTF
ie, + y
25, 959
MIMIC
LF
(t,ta] -
262
mortality, ICD-9 codes
FLICKR-300 30K
VILT
Es
158, 000
image-caption retrieval
Retrieval
FLICKR-30K
CLIP
8++
158, 000
image-caption retrieval
CLEVR
CNN-LSTM-SA MDETR
[1, y
853, 554
QA QA QA
QA
CLEVR VQA 2.0
[1, + y
853, 554
LXMERT
[i,2] + y
1, 100, 000
Setup: We use a large suite of datasets from MultiBench (Liang et al., 202 la) which span world fusion (Zadeh et al., 2018; Arevalo et al., 2017: Johnson et al., 2016), retrieval (Plummer et al., 2015) 5), and QA (Johnson et al., 2017: Goyal et al., 2017) tasks. For each dataset, we test a corresponding state-of-the-art model: MULT (Tsai al., 2019), LRTF (Liu et al., 2018), LF (Baltrušaitis al., 2018), VILT (Kim et al., 2021), CLIP (Radford er al., 2021), CNN-LSTM- A (Johnson et al., 2017), MDETR (Kamath al., 2021), and LXMERT (Tan and Bansal, 2019). These cover models both pretrained and trained from scratch. We summarize all 6 datasets and 8 models tested in Table 1, and provide implementation details in Appendix C and user study details in Appendix D.
3.1 MODEL SIMULATION
We first design model simulation experiment to determine if MULTIVIZ helps users of multimodal models gain a deeper understanding of model behavior. If MULTIVIZ indeed generates human- understandable explanations, humans should be able to accurately simulate model predictions given these explanations only, as measured by correctness with respect to actual model predictions and annotator agreement (Krippendorff's 's alpha (Krippendorff, 2011)). To investigate the utility of each stage in MULTIVIZ, we design human study to see how accurately 21 humans users (3 users for each of the following local ablation settings) can simulate model predictions:
(1) U: Users are only shown the unimodal importance (U) of each modality towards label y.
(2) U + C: Users are also shown cross-modal interactions (C) highlighted towards label y.
(3) U + C+ R₁: Users are also shown local analysis (Rş) of unimodal and cross-modal interactions
of top features Zuop [2(1). .(2)...) maximally activating label y.
(4) U+ C+R + Ry: Users are additionally shown global analysis (R,)) through similar datapoints that also maximally activate top features Zwo for label y.
(5) MULTIVIZ (U C+ R R, P): The entire MULTIVIZ method by further including visualiza- tions of the final prediction (P) stage: sorting top ranked feature neurons zop = (2(1)- =..)....} with
respect to their coefficients Srop = {8(1), B(2) ...) and showing these coefficients the user.
5

---

Published as a conference paper at ICLR 2023
Table 2: Model simulation: We tasked 15 humans users (3 users for each of the following local ablation settings) simulate model predictions based on visualized evidences from MULTIVIZ. Human annotators who have access to all stages visualized in MULTIVIZ are able to accurately and consistently simulate model predictions (regardless of whether the model made the correct prediction) with high accuracy and annotator agreement, representing step towards model understanding.
Research area
QA VQA 2.0 LXMERT
Fusion MM-IMDB LRTF
Fusion CMU-MOSEL MULT
Dataset
Model
Metric
Correctness
Agreement
Correctness
Agreement
Correctness
Agreement
U U+C
55.0 0.0
0.39
50.0 + 13.2
0.34
71.7 + 17.6
0.39
65.0 + 5.0
0.50
53.7 7.6
0.51
76.7 10.4
0.45
U+C R
61.7 + 7.6
0.57
56.7 1 7.6
0.59
78.3 1 2.9
0.42
U+C+R, R. + Rg
71.7 1: 15.3
0.61
61.7 1 7.6
0.43
100.0
0.0
1.00
MULTIVIZ
81.7 1 2.9
0.86
65.0 1 5.0
0.60
100.0
11.00-100 0.0
1.00
Using 20 datapoints per setting, these experiments with 15 users on datasets and models involve 35 total hours of users interacting with MULTIVIZ, which is significantly larger-scale study of model simulation compared to prior work (Aflalo et al., 2022; Lyu et al., 2022; Wang et al., 2021).
Quantitative results: We show these results in Table 2 and find that having access to all stages in MULTIVIZ leads to significantly highest accuracy of model simulation on VQA 2.0, along with lowest variance and most consistent agreement between annotators. On fusion tasks with MM- IMDB and CMU-MOSEI, we also find that including each visualization stage consistently leads to higher correctness and agreement, despite the fact that fusion models may not require cross-modal interactions to solve the task (Hessel and Lee, 2020). More importantly, humans are able to simulate model predictions, regardless of whether the model made the correct prediction not.
To test additional intermediate ablations, we conducted user studies on (6) R. P (local analysis on final-layer features along with their prediction weights) and (7) Rg P (global analysis on final-layer features along with their prediction weights), to ablate the effect of overall analysis (U and C) and feature analysis (R, or Rg in isolation). R, P results in an accuracy of 51.7 + 12.6 with 0.40 agreement, while R, + P gives 71.7 + 7.6 with 0.53 agreement. Indeed, these underperform as compared to including overall analysis (U and C) and feature analysis (R, R2).
Finally, we also scaled 100 datapoints on VQA 2.0, representing upwards of 10 hours of user interaction (for the full MULTIVIZ setting). and obtain an overall correctness of 80%, reliably within the range of model simulation using 20 points (81.7 + 2.9). Therefore, the sample size of 20 points that makes all experiments feasible is still reliable sample.
We also conducted qualitative interviews to determine what users found useful in MULTIVIZ:
(1) Users reported that they found local and global representation analysis particularly useful: global analysis with other datapoints that also maximally activate feature representations were important for identifying similar concepts and assigning them to multimodal features.
(2) Between Overview (U + C) and Feature (R, + R. P) visualizations, users found Feature visualizations more useful in 31.7%, 61.7%, and 80.0% of the time under settings (3), (4), and (5) respectively, and found Overview more useful in the remaining points. This means that for each stage, there exists significant fraction of data points where that stage is most needed.
(3) While may be possible determine the prediction of the model with subset of stages, having more stages that confirm the same prediction makes them a lot more confident about their prediction, which is quantitatively substantiated by the higher accuracy, lower variance, and higher agreement in human predictions. We also include additional experiments in Appendix D.
3.2 REPRESENTATION INTERPRETATION
We now take deeper look to check that MULTIVIZ generates accurate explanations of multimodal representations. Using local and global representation visualizations, can humans consistently assign interpretable concepts in natural language previously uninterpretable features? We study this question by tasking 15 human users (5 users for each of the following 3 settings) to assign concepts to each feature when given access to visualizations of (1) R, (local analysis of unimodal and cross-modal interactions in 2), (2) R, + R, (no viz) (including global analysis through similar datapoints that also maximally activate feature 2), and (3) R2 Rg (adding highlighted unimodal and

---

Published as a conference paper at ICLR 2023
Table 3: Left: Across 15 human users (5 users for each of the following 3 settings), we find that users able consistently assign concepts previously uninterpretable multimodal features using both local and global representation analysis. Right: Across 10 human users (5 users for each of the following settings), find that users are also able categorize model errors into one of 3 stages they occur in when given full MULTIVIZ visualizations.
Research area
QA
Research area
QA CLEVR CNN-LSTM-SS
QA VQA 2.0
Dataset
VQA 2.0 LXMERT
Dataset
Model
Model
LXMERT
Metric
Confidence
Agree.
Metric
Confidence
Agree.
Confidence
Agree.
Rr
1.74 0.52
0.18
No viz
2.72 1 0.15
0.05
2.15 1 0.70
0.14
Rr + Ry (no viz)
3.67 + 0.45
0.60
MULTIVIZ
4.12 1 0.45
0.67
4.21 0.62
0.60
R Rg
4.50 0.43
0.69
Image concept: holding sports equipment (VQA 2.0)
Multimodal concept: asking about material (VQA 2.0)
What this person have on their fore?
What color the bot?
What painted on the ground under the racket?
counter made of?
What countertop mode of?
made of?
Language concept: positive mentions text (CMU-MOSEI)
Multimodal concept: animals detected (MM-IMD6)
BORK
Over 50% fandea! That's amozing!
Nove you! Are you saying..
I'd like to inite you to the..
very little Pollution unyellded
approaches maturity... broin, his friends.. evil monster Dogahra..
Figure 4: Examples of human-annotatee concepts using MULTIVIZ on feature representations. We find that the features separately capture image-only, language-only, and multimodal concepts.
cross-modal interactions of global datapoints). Using 20 datapoints per setting, these experiments with 15 users involve roughly 10 total hours of users interacting with MULTIVIZ..
Quantitative results: Since there are ground-truth labels for feature concepts, we rely on annotator confidence (1-5 scale) and annotator agreement (Krippendorff, 2011) as a proxy for accuracy. From Table 3 (left), we find that having access to both local and global visualizations interpreting multimodal features, as measured by higher confidence with low variance in confidence, well higher agreement among users.
crucial towards
Qualitative interviews: We show examples of human-assigned concepts in Figure 4 (more in Appendix D.3). Note that the 3 images in each box of Figure (even without feature highlighting) does constitute visualization generated by MULTIVIZ, as they belong to data instances that maximize the value of the feature neuron (i.e. Rg in stage multimodal representations) Without MULTIVIZ it would not be possible to perform feature interpretation without combing through the entire dataset. Participants also noted that feature visualizations make the decision a lot more confident if its highlights match the concept. Taking as example Figure top left, the visualizations serve to highlight what the model's 's feature neuron is learning (i.c., highlighting the person holding sports equipment), rather than what category of datapoint is. If the visualization was different, such as highlighting the ground, then users would have to conclude that the feature neuron is capturing "outdoor ground' rather than 'sports equipment'. Similarly, for text highlights (Figure 4 top right), without using MULTIVIP to highlight 'counter', 'countertop', and "wall", along with the image crossmodal interactions corresponding to these entities, one would not be able to deduce that the feature asks about material it could also represent 'whul' questions, or 'household objects', and so on. Therefore, these conclusions can only reliably deduced with all MultiViz stages.
3.3 ERROR ANALYSIS
We further examine case study of error analysis trained models. We task 10 human users (5 users for each the following 2 settings) to use MULTIVIZ and highlight the errors that a multimodal model exhibits by categorizing these errors into one of 3 stages: failures in (1) unimodal perception, (2) capturing cross-modal interaction, and (3) prediction with perceived unimodal and cross-mou cross-modal information. Again, rely annotator confidence (1-5 scale) and agreement due lack of
7

---

Published as a conference paper at ICLR 2023
Unimodal perception errors
Cross-modal interaction errors
Prediction errors
From visuaiziing unimedal importonce, model detect the streak.
From visualiving crass- model interactions, the model foils capture the interartion between 'creait and the image.
From visuplizing cross- model interactions, the model deteets uccurately, but micclssaftes its material the reasoning level.
What the Pred: white. Correct: red
a creamy soup? Pred: yes, Correct: No
Whet
of? Correct: leather
Pred: plastic,
From visuallaing unimedoi importance, the model falls detect
From visualleing cross- modol interactions the model the interaction between "targe brown matte thing' image.
From visualzing cross- modal Interactions, the mosel detects the other cyllnders accurately, but mizunderstand: the word "ocher" and did not exclude the original object.
Image.
The Calffannia Angels one currently the their divisio...
What that is the some shope as the large brown motte thing?
object
What of other things are the some shape as the purple motte thing? Correct.
Pred: comedy, Correct: sport
Pred: gray. brown
Figure 5: Examples of human-annotated error analysis using MULTIVIZ on multimodal models. Using all stages provided in MULTIVIZ enables fine-grained classification of model errors (c.g.. errors in unimodal processing, cross-modal interactions, and predictions) for targeted debugging. (unimodal)
(unimodal)
(cross-modal)
(cross-modal)
What the plastic bins? Predicted: orange. Correct: blue
What color is the cone?
Predicted: blue. Correct: orange
Figure 6: A case study on model debugging: we task 3 human users to use MULTIVIZ visualizations and highlight the errors that a pretrained LXMERT model fine-tuned on VQA 2.0 exhibits, and find 2 penultimate-layer neurons highlighting the model's failure to identify color (especially blue). Targeted localization of the error this specific stage (prediction) and representation concept (blue) via MULTIVIZ enabled us to identify a bug in the popular Hugging Face LXMERT repository.
ground -ruuh error categorization, and compare (1) MULTIVIZ with (2) No viz. a baseline that does not provide any model visualizations to the user. Using 20 datapoints per setting, these experiments with 10 users on 2 datasets and models involve roughly 15 total hours of users interacting with MULTIVIZ. From Table 3 (right), we find that MULTIVIZ enables humans to consistently categorize model errors into one of 3 stages. We show examples that human annotators classified into unimodal perception, cross- modal interaction, and prediction errors in Figure (more in Appendix D.4).
3.4 A CASE STUDY IN MODEL DEBUGGING
Following error analysis, we take a deeper investigation into one of the errors on a pretrained LXMERT model fine-tuned on VQA 2.0. Specifically, first found the top 5 penultimate- neurons that are most activated on erroneous datapoints. Inspecting these neurons carefully through MULTIVIZ local and global representation analysis, human annotators found that 2 of the neurons were consistently related to questions asking about color, which highlighted the model's failure to identify color correctly (especially blue). The model has an accuracy of only 5.5% amongst all blue-related points (i.e., either have blue as correct answer or predicted answer). and these failures account for 8.8% of all model errors. We show examples of such datapoints and their MULTIVIZ visualizations in Figure 6. Observe that the model is often able to capture unimodal and cross-modal interactions perfectly, but fails identify color at prediction.
Curious the source of this error, looked deeper into the source code for the entire pipeline of LXMERT, including that of its image encoder, Faster R-CNN (Ren et al., 2015)'. We in fact uncovered a bug in data preprocessing for Faster R-CNN in the popular Hugging Face repository that swapped the image data storage format from RGB to BGR formats responsible for these errors. This presents a concrete use case MULTIVIZ: through visualizing each stage, we were able to (1) isolate the source of the bug (at prediction and not unimodal perception or cross-modal interactions), and (2)
'w used the popular Hugging Face implementation at https: //huggi facace. co/unc-nlp/ xmert comeert-va--te vgaaundased
8

---

Published as a conference paper at ICLR 2023
use representation analysis to localize the bug to the specific color concept. In Appendix D.5, further detail our initial attempt at tackling this error by using MULTIVIZ analysis to select additional targeted datapoints in an active learning scenario, which proved to be much more effective (higher improvement with fewer data) as compared to baselines that add data randomly or via uncertainty sampling (Lewis and Catlett, 1994), which may be of independent interest.
3.5 ADDITIONAL. EXPERIMENTS AND TAKEAWAYS MESSAGES
New models: We included results on VILT (Kim et al., 2021), CLIP (Radford et al., 2021), and MDETR (Kamath et al., 2021) in Appendix D.2, showing that MULTIVIZ is general approach that can be quickly applied to new models. We also study the correlation between performance and cross- modal interactions across several older and recent models, and find that the ability to capture cross- modal alignment, as judged by MULTIVIZ, correlates strongly with final task performance.
Sanity checks: In Appendix A.5, we show that MULTIVIZ passes the data randomization and model randomization sanity checks for interpretation approaches (Adebayo et al., 2018).
Intermediate-layer features: In Appendix B.3, we show that MULTIVIZ can be extended to visualize any intermediate layer, not just the final layer of multimodal models. We showcase a few examples of R₂ and R, on intermediate- -lyee neurons and discuss several tradeoffs: while they reveal new visualization opportumities. they run the risk of overwhelming the user with the number of images they have to see multiplied by d= (d: dimension of each layer, L: number of layers).
just
RELATED WORK
Interpretable ML aims to further our understanding and trust of ML models, enable model debugging. and use these insights for joint decision decision-making between stakeholders and Al (Chen et al., 2022; Gilpin et al., 2018). Interpretable MI. is a critical area of research straddling machine learning (Ade- bayo et al., 2018), language (Tenney et al., 2020), vision (Simonyan et al., 2013), and HCI (Chuang et al., 2012). We categorize related work in interpreting multimodal models into:
Unimodal importance: Several approaches have focused on building interpretable components for unimodal importance through soft (Park et al.. 2018) and hard attention mechanisms (Chen al., 2017). When aiming to explain black-box multimodal models, related work rely primarily on gradient- visualizations (Simonyan et al., 2013; Baehrens et al., 2010; Erhan et al., 2009) and feature attributions (e.g., LIME (Ribeiro et al., 2016). Shapley values (Merrick and Taly, 2020)) to highlight regions of the image which the model attends to.
Cross-modal interactions: Recent work investigates the activation patterns of pretrained transform- ers (Cao et al., 2020; Li et al., 2020), performs diagnostic experiments through specially curated inputs (Frank et al., 2021; Krojer et al., 2022; Parcalabescu et al., 2021; Thrush et al., 2022), or trains auxiliary explanation modules (Kanehira et al., 2019; Park et al., 2018). Particularly related to our work is EMAP (Hessel and Lee, 2020) for disentangling the effects of unimodal (additive) contributions from cross-modal interactions in multimodal tasks, as well as M2Lens (Wang et al., 2021), an interactive visual analytics system to visualize multimodal models for sentiment analysis through both unimodal and modal contributions.
Existing approaches have used language syntax (e.g.. the question in VQA) for compositionality into higher-level features (Amizadeh al., 2020; Andreas et al., 2016; Vedantam et al., 2019). Similarly, logical statements have been integrated with neural networks for interpretable logical reasoning (Gokhale et al., 2020: Suzuki al., 2019). However, these are typically restricted to certain modalities or tasks. Finally, visualizations have also uncovered several biases in models and datasets (e.g., unimodal biases in VQA questions (Anand al., 2018; Cadene et al., 2019) or gender biases in image captioning (Hendricks et al., 2018)). We believe that MULTIVIZ will enable the identification of biases across a wider range of modalities and tasks.
Multimodal representation and prediction:
5 CONCLUSION
This paper proposes MULTIVIZ for analyzing and visualizing multimodal models. MULTIVIZ scaffolds the interpretation problem into unimodal importance, cross-modal interactions, multimodal representations, and multimodal prediction, before providing existing and newly proposed analysis tools in each stage. MULTIVIZ is designed be modular (encompassing existing analysis tools and encouraging research towards understudied stages), general (supporting diverse modalities, models, (providing a visualization tool for human model interpretation. error analysis, and debugging). qualities which we strive to upkeep by ensuring its public access and regular updates from community feedback.
and tasks), and human-in-ther
9

---

Published as a conference paper at ICLR 2023
6 ETHICS STATEMENT
Multimodal data and models are ubiquitous in range of real-world applications. MULTIVIZ is our attempt at a standardized and modular framework for visualizing these multimodal models. While we believe these tools can help stakcholders gain deeper understanding and trust of multimodal models as a step towards reliable real-world deployment, we believe that special care must be taken in the following regard to ensure that these tools are reliably interpreted:
1. Reliability of visualizations: There has been recent work examining the reliability of model interpretability methods for weal-world practitioners (Pruthi et al., 2020; Srinivas and Fleuret, 2020). Lipton (2018) examines the motivations underlying interest in interpretability, finding them to be diverse and occasionally discordant. Krishna al. (2022) find that state-of-the-ar art explanation methods may disagree in terms of the explanations they output. Chandrasekaran et al. (2018) further conclude that existing explanations on VQA model do not actually make its responses and failures more predictable to a human. We refer the reader to Chen et al. (2022) for critique on the disconnect between technical objectives targeted by interpretable ML research and the high-level goals stated as consumers' use cases, well Bhatt al. (2020) for analysis of how interpretable and explainable ML tools can be used in real-world deployment. Human- in-the-loop interpretation and evaluation could be a promising direction towards connecting technical solutions with real-world stakeholders, while also offering users an interactive medium to incorporate feedback in multimodal models.
2. Pitfalls of gradient-based interpretation: We are aware of the limitations underlying gradient- based interpretation of black-box -bx models (Lipton, 2018; Srinivas and Fleuret, 2020) with issues surrounding their laithfulness and usefulness. Future work should examine the opportunities and risks of gradient- pradient-based approaches. particularly in the context of cross-modal interactions.
3. The role of rros--modal interactions: There has been work showing that certain multimodal tasks do not need models to pick up cross-modal interactions to achieve good performance (Hessel and Lee, 2020). Indeed, for tasks like cross-modal retrieval, simply learning one interaction between a word and its corresponding image region is enough for typical datasets. This makes interpretation of cross-modal interactions difficult, since even well-performing models may not need to pick up all cross-modal interactions.
4. User studies: Based on direct communication with our institution's 's IRB office, this line of user-study research is aligned with similar annotation studies at our institution that are exempt from IRB. The information obtained during our study is recorded in such a manner that the identity of the human subjects cannot readily be ascertained, directly or through identifiers linked to the subjects. We do not collect any identifiable information from annotators.
5. Usability: While we tried to be comprehensive in providing visualizations to the user, more information beyond certain point is probably not useful and may overwhelm the user. We plan to work closely with HCI researchers to rethink usability and design of our proposed interpretation tools through careful user studies. MULTIVIZ will also welcome feedback from the public to improve its usability.
6. Beyond MULTIVIZ stages: While we believe that many multimodal problems can benefit from breaking them down into our proposed interpretation stages, we also acknowledge that certain problems may not benefit from this perspective. For example, problems in multimodal translation (mapping from modality another, such as image captioning) will not involve prediction layers and instead require new stages to interpret the generation process, and problems in cross- modal transfer will also require new stages interpret knowledge transfer. In Appendix E, we include more details on new datasets we plan to add to MULTIVIZ to enable the study of new multimodal interpretability problems, and other interpretation tools we plan to add.
7. Evaluating interpretability: Progress towards interpretability is challenging to evaluate (Chan et al., 2022; Dasgupta et al., 2022; Jacovi and Goldberg, 2020; Shah et al., 2021; Srinivas and Fleuret, 2020). Model interpretability (1) is highly subjective across different population subgroups (Arora et al., 2021; Krishna et al., 2022), (2) requires high-dimensional model outputs as opposed to low-dimensional prediction objectives (Park et al., 2018), and (3) has desiderata that change across research fields, populations, and time (Murdoch et al., 2019). We plan to continuously expand MULTIVIZ through community inputs for new interpretation methods in each stage and metrics to evaluate interpretability methods (see Appendix E for details). Some metrics we have in mind include those for measuring faithfulness, as proposed in recent work (Chan et al., 2022; Dasgupta et al., 2022; Jacovi and Goldberg, 2020; Madsen et al., 2021; Shah et al., 2021: Srinivas and Fleuret, 2020; Wu and Mooney, 2019).
10

---

Published as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
1. Our code, datasets, and documentation are released at https:/ com/ pliang279 /t///Mulliiiiii This link also includes human-in-the- in-the-loopp evaluation scripts and instructions on running MULTIVIZ for new datasets and models.
2. Details on the MULTIVIZ visualization approaches are provided in Appendix A.
3. Details on the MULTIVIZ website, sample webpages with visualizations, code structure, and sample tutorials are provided in Appendix B.
4. Dataset collection and preprocessing details are provided in Appendix C. We provide documentation for MULTIVIZ in the form of datasheets for datasets (Gebru et al., 2018).
5. Experimental details, including all details on user studies and evaluation, are provided in Appendix D.
ACKNOWLEDGEMENTS
This material based upon work partially supported by the National Science Foundation (Awards #1722822 and #1750439), National Institutes of Health (Awards #R01MH125740,, #ROIMH096951, and #U0IMH1 1692)), and BMW of North America. PPL is partially supported by a Facebook PhD Fellowship and a Carnegie Mellon University's Center for Machine Learning and Health Fellowship. RS is supported in part by ONR award N000141812868 findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, National Institutes of Health, Facebook, Carnegie Mellon University's Center for Machine Learning and Health, Office of Naval Research, or DSTA, and no official endorsement should be inferred. We are extremely grateful to Ben Eysenbach, Martin Ma, Chaitanya Ahuja, Volkan Cirik, Peter Wu, Amir Zadeh, Alex Wilf, Victoria Lin, Dong Won Lee, Torsten Wortwein, and Tiffany Min for helpful discussions and feedback on initial versions of this paper. Finally, we would also like to acknowledge NVIDIA's GPU support.
and DSTA. Any opinions,
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. CoRR, abs/1810.03292, 2018, URL http:://axxiiii org. haagg//bs/18/ 03292. 9, 19, 20
Estelle Aflalo, Meng Du Shao- Yen Tseng. Yongfei Liu. Chenfei Wu. Nan Duan, and Vasudev Lal. VI- interpret: An interactive visualization tool for interpreting vision-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2140-1144,5 2022. 6, 49
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. VQA: Visual question answering. International Journal of Computer Vision, 2017. 30
Saeed Amizadeh, Hamid Palangi, Alex Polozov, Yichen Huang, and Kazuhito Koishida. Neuro-symbolic visual reasoning: Disentangling. In International Conference on Machine Learning, pages 279-290 PMLR, 2020.
Ankesh Anand, Eugene Belilovsky, Kyle Kastuer, Hugo Larochelle, and Aaron Courville. Blindfold baselines embodied qa. arXiv preprint arXiv:1811.000 1811,0001. 2018.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 39-48, 2016. 49
John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. In 5th International conference on learning representations 2017 workshop, 2017. 29
Siddhant Arora, Danish Pruthi, Norman Sadeh, William W Cohen, Zachary C Lipton, and Graham Neubig. Explain, edit, and understand: Rethinking user study design for evaluating model explanations. arXiv preprint arXiv:2112. 09669, 2021. 10,
David Baehrens, Timon Schroeter. Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Robert Müller. How explain individual classification decisions. The Journal of Machine Learning Research. 11: 1803-1831, 2010.
18, 20
Tadas BaltruKaitis, Chaitanya Ahuja, and Louis- Morency. Multimodal machine learning: A survey and taxonomy. IEFE transactions on pattern analysis and machine intelligence, 41(2):23-443 2018. 5
11

---

Published as a conference paper at ICLR 2023
Umang Bhatt, Alice Xiang. Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. Explainable machine learning deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages (18-657, 2020. 10, 49
Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. Advances in Neural Information Procesiing Systems, 32-841-852, 2019. 9
Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained vision-and -nd--anddd-daa languag models. In European Conference on Computer Vision, pages 565 580. Springer, 2020. 9
Chun Sik Chan, Huanqi Kong, and Liang Guanqing. A comparative study of faithfulness metries for model interpretability methods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisticx (Volume Long Pupers), pages 5029-5038, 2022. 10, 49
Arjun Chandrasckaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh, Do explanations make vqa models more predictable to human? In EMNLP, 2018. 10, 18
Minghai Chen, Sen Wang. Paul Pu Liang, Tadas Baltrušaitis, Amir Zadeh, and Louis-Philippe Morency. Multimodal sentiment analysis with word-level fusion and reinforeement learning. In Proceedings of the 19th ACM International Conference Multimodal Interaction, pages 163-171 171, 2017.
Valerie Chen, Jeffrey Li, Joon Sik Kim, Gregory Plumb, and Ameet Talwalkar. Interpretable machine learning: Moving from mythos to diagnostics. Queue, 19(6):28 56, 2022. 9, 10, 49
Jason Chuang, Daniel Ramage, Christopher Manning, and Jeffrey Heer. Interpretation and trust: Designing model-driven visualizations for text analysis. In Proceedings of the SIGCHI conference on human factors computing systems, pages
2012. 9
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra, Emhodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-10, 2018. 30
Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. Framework for evaluating faithfulness of local explana- tions. arXiv preprint arXiv:2202.000 00734, 2022. 10, 49
Daisy Yi Ding and Robert Tibshirani. Cooperative learning for multi-view analysis. arXiv:2112.12332 2021.
arXiv preprint
Stéphanc Dupont and Juergen Lucttin. Aisual speech modeling for continuous specch recognition. JEEE transactions on multimedia, 2(3):111 151, 2000. 29
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009. 2, 9, 18, 20
Stella Frank, Emanuele Bugliarello, and Desmond Elliou. Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages
2021. 9
Jerome H Friedman and Bogdan E Popescu. Predictive leamning via rule ensembles. The annals of applied statistics, (33)916-954. 2008. 3, 18
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. arXiv preprint arX3v: 090110 2018. 11
Leilani H Gilpin. David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretabiliity machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pages 80-89. IEEE, 2018. 9
Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Vqa-lol: Visual question answering under the lens of logic. In European conference on computer vision, pages 379-396. Springer, 2020. 9
Yash Goyal, Akrit Mohapatra, Devi Parikh, and Dhruv Batra. Towards transparent ai systems: Interpreting
visual question answering models. arXiv preprint 088794160..0897,, 2016. 1, 19
Yash Goyal, Tejas Khot, Douglas Summers-Stee Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 6904-6913, 2017. 30, 46, 47
12

---

Published as a conference paper at ICLR 2023
Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3353-5563, 2020, 3, 18, 20
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In Proceedings of the European Conference on Computer Vision (ECCV), pages 771 771-787, 2018. 1,9
Lisa Anne Hendrrckss, John Mellor, Rosalia Schneider. Jean-Baptiste Alayrac, and Aida Nematzadeh. Decoupling role data, attention, and losses in multimodal transformers. Transactions of the Association for Computational Linguistics, 9:570-588 585, 2021. 2
Jack Hessel and Lillian Lee. Does my multimodal model learn cross cross-modal interactions? it's harder to tell than you might think! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLP), pages 861 2020. 1, 3, 6, 9, 10, 18, 20
Ronghang Hu and Amanpreet Singh. Transformer is all you need: Multimodal multitask learning with unified transformer. arXiv preprint arXiv:2102.10772. 2021.
Allan Jabri, Armand Joulin, and Laurens van der Maaten. Revisiting visual question answering baselines. In European conference on computer vision, pages 727-739. Springer, 2016.
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198-4205, 2020. 10, 49
Siddhant M Jayakumar, Wojeiech M Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where find them. In International Conference on Learning Representations, 2020. 3
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016. 29
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017. 5, 30, 38
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion, Mdetr- modulated detection for end-to-end multi-modal understanding. arXiv preprint 1rXiv:2104.111 12763, 2021. 5, 9,
38
Atsushi Kanehira, Kentaro Takemoto, Sho Inayoshi, and Tatsuya Harada. Multimodal explanations by predicting counterfactuality videos. In Proceedings of the IEEF/CVP Conference on Computer Vision and Paternn Recognition, pages 8594-8602 2019." 9
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. arXiv preprint McXiv:1909.029953 2019. 29
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languaag transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583-5594. PMI.R, 2021. 5, 39
Klaus Krippendorff. Computing krippendorff's alpha-reliabilit 2011. 5.7
Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitioner's perspective. arXiv preprint arXiv:220..01602, 2022. 10, 49
Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. Image retrieval from contextual descriptions. Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume Long Papers), pages 1426-3440, 2022.
Michelle A Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei. Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Self Self- supervised learning of multimodal representations for contact-rich tasks. In 20/9 International Conference Robotics and Automation (ICRA), pages 8943-8950. IEFF, 2019.
13

---

Published as a conference paper at ICLR 2023
David D Lewis and Jason Catlett. Heteeoggeneous uncertainty sampling for supervised learning. In Machine learning proceedings 1994, pages 148- 156. Elsevier, 1994, 9, 47
David D Lewis and William A Gale. A sequential algorithm for training text classifiers. In SIGIR 94, pages
3-12. Springer, 1994. 47
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.033330 2019. 1,2
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Wei Chang. What does bert with vision look at? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5265-5275, 2020. 9
Paul Pu Liang, Ziyin Liu, AmirAli Bagher Zadeh, and Chiil-Philippe Morency. Multimodal language analysis with recurrent multistage fusion. In EMNLP, 2018a. 29
Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philipee Morency. Computational modeling of human multimodal language: The mosei dataset and interpretable dynamic fusion. 2018b. 29,
Paul Pu Liang. Yiwei I Lu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee. Yuke Zhu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Multibench: Multiscale benchmarks for multimodal representation learning. NeurlPS Datasets and Benchmarks Track, 2021a. 29, 32, 36
Paul Pu Liang, Peter Wu, Liu Ziyin, Louis-Philippe Morency, and Ruslan Salakhutdinov. Cross-modal gen- eralization: Learning in low resource modalities via meta-al alegmalli.mmn.. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2680-20000 2009226892220 2021b.
Paul Pu Liang, Amir Zadeh, and Louis-Philipee Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint (XXi:22009004388 2022. 1
Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is
both important and slippery. Queue, 16(3):31-57, 2018. 10
Zhun I. Liu, Ying Shen, Varun Bharadhwaj I lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadch, and Mouis-Philippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2247-2256, 2018.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: pretraining task-agnostic visiolinguistic repre- sentations for vision-and aooon-and-lamee lan-uang tasks. In Proceedings of the 33rd International Conference on Neurul Information Processing Systems, pages 13-23, 2019. 1.2, 49
Yiwei Lyu, Paul Pu Liang. Zihao Deng, Ruslan Salakhutdinov. and Louis Phill-ph Morency. Dime: Fine- Fineigrained interpretations of multimodal models via disentangled local explanations. arXiv preprint arXiv:2203.02013. 2022. 3, 6, 18, 20
Andreas Madsen, Nicholas Meade, Vaibhav Adlakha, and Siva Reddy. Evaluating the faithfulness of impor- tance measures in nip by recursively masking allegedly important tokens and retraining. arXiv preprint arXiv:2110.08412, 2021. 10, 49
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In International Conference on Learning Representations, 2019. URL https: : /ttp///vv////../////////// net http///n/////eeee 48
Luke Merrick and Ankur Taly. The explanation game: Explaining machine learning models using shapley values. In International Cross-Domain Conference for Machine Learning and Knowledge Extraction, pages 17-38. Springer, 2020. 3, 9, 18, 19,
W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences, 116(44); 22071-22080 2019. 10, 49
Milind Naphade, John R Smith, Jelena Tesic, Shih-Fu Chang, Winston Hsu, Lyndon Kennedy, Alexander Hauptmann, and Jon Curtis. Large-scale concept ontology for multimedia. IEEE multimedia, 13(3);6-91, 2006. 1
Letitia Parcalabescu, Albert Gatt, Anette Frank, and lacer Calixto. Seeing past words: Testing the cross-modal capabilities of pretrained v&l models on counting tasks, In Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR), pages 32-44, 2021. 9
14

---

Published as a conference paper at ICLR 2023
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8779-87888 8788, 2018. 1, 9, 10, 49
Man-MM Pérez-Rías Rúa, Valentin Vielzeuf, Stéphane Pateux, Moez Baccouche, and Frédéric Jurie. Mfas: Multimodal fusion architecture search. In Proceedings of the IEEE Conference computer vision and pattern recognition, pages
2019. 29
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-piin phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 1249, 2015. 5, 30, 39
Soujanya Poria, Erik Camhria, Rajiv Bajpai, and Amir Hussain. A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 2017. 1
Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C Lipton. Learning deceive with attention-based explanations. In Proceedings the 58th Annual Meeting of the Association for Computational Linguisticx, pages 4782-4793, 2020. 10
Sanjay Purushotham, Chuizheng Meng, Zhengping Che, and Yan Liu. Benchmarking deep learning models on large healthcare datasets. Journal of Biomedical Informatics, 83:112-134, 2018. ISSN 1532-0464. doi: http:///doi.og110..110000 jii Utthttts: / /wwww aci encedi rect.com/ com/ aci ence/ article/ aetticeeeppi/S1132441111111 /tt.///1111200111011111111113 29
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021. 5,
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015. 8
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should trust you?" explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135 144, 2016. 1, 3, 9, 18, 19, 20
Raquel Rodriguez- Pérez and Jürgen Bajorath. Interpretation of machine learning models using shapley values: application to compound potency and multi-targe activity predictions. Journal of computer-aided molecular design, 34(10):0013-1026 2020. 18, 20
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-baseo localization, Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017. 18
Burr Settles. Active learning literature survey. 2009. 47
Harshay Shah, Prateck Jain, and Pranceth Netrapalli. Do input gradients highlight discriminative features? Advances in Neurul Information Processing Systems, 34, 2021. 10, 49
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: 1312.60344 2013. 9, 18, 20
Daria Sorokina, Rich Caruana. Mirek Riedewald. and Daniel Fink. Detecting statistical interactions with additive groves of trees. Proceedings of the 25th international conference on Machine learning, pages 1000-1007, 2008. 3
Suraj Srinivas and Francois Fleuret. Rethinking the role of gradient- attribution methods for model interpretability. In International Conference on Learning Representations, 2020. 10, 49
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In International conference on machine learning, pages 9269-92999 -978. PMLR. 2020.
Riko Suzuki, Hitomi Yanaka, Masashi Yoshikawa, Koji Mineshima, and Daisuke Bekki. Multimodal logical inference system for visual-textual entailment. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 386-392, 2019. 9
Hao Tan and Mohit Bansal. LXMERT: learning cooss-modality encoder representations from transformers. CoRR, abs/1908 2019. http: wwtp///wwwwg///////////ooooos 9988 07490. 46, 47
15

---

Published as a conference paper at ICLR 2023
Ian Tenney, James Wexler, Jasmijn Bastings. Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann. Ellen Jiang, Mahima Pushkama, Carcy Radebaugh, Emily Reif, et The language interpretability tool: Extensible, interactive visualizations and analysis for nip models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lunguage Processing: System Demonstrations, pages 107-118, 2020. 9
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2022. 9
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang. J Zico Kolter, Louis-Philipee Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational I Linguistics, pages 6558-6569, 2019. 2,5
Michael Tsang, Dehua Cheng, and Yan Liu, Detecting statistical interactions from neural network weights. In International Conference on Learning Representations, 2018.
Michael Tsang, Dehua Cheng, Hanpeng Liu, Xue Feng, Fric Zhou, and Yan Liu. Feature interaction interpretabil- ity: A case for explaining ud-recommendation systems via neural interaction detection. In International Conference on Learning Representations, 2019. 3, 18
Ramakrishna Vedantam, Karan Desai, Stefan Lee, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Probabilistic neural symholic models for interpretable visual question answering. International Conference on Machine Learning, pages 6428-6437, 2019. 9
Alvin Wan, Lisa Dunlap, Danicl Ho, Jihan Yin, Scott Lec, Suzanne Petryk, Sarah Adel Bargal, and Joseph Gonzalez. Nbdt: Neural-backed decision tree. In International Conference on Learning Representations, 2020. 49
Xingbo Wang, Jianben Hc, Zhihua Jin, Muqiao Yang, Yong Wang, and Huamin Qu. M2lens: Visualizing and explaining multimodal models for sentiment analysis. IEEE Trunsactions on Visualization and Computer Graphicx, 28(1):80-8112, 2021. 6,949
Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In International Conference Machine Learning, pages 11205-11216. PMLR, 2021. 4, 19
Jialin Wu and Raymond Mooney. Faithful multimodal explanation for visual question answering. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 103-112 112-112, 2019. 10
Keyang Xu, Mike I Lam, Jingzhi Pang, Xin Gao, Charlotte Band, Piyush Mathur, Frank Papay, Ashish Khanna, Jacek B Cywinski. Kamal Maheshwari, et al. Multimodal machine learning for automated ied coding. In Machine Learning for Healthcare Conference, pages 197-215. PMLR, 2019. I
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 10:110310000 2018. 49
Jason Yosinski, Jeff Clune, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In In ICML Workshop on Deep Learning. Citeseer, 2015. 3, 18, 20
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 8807-8817, 2019. 30
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mose dataset and interpretable dynamic fusion graph. In ACL, 2018. 5. 29,
Lingl Zhen, Peng Hu, Xu Wang, and Dezhong Peng. Deep supervised cross-modal retrieval. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10394-1044 10403, 2019.
16

---

Published as a conference paper at ICLR 2023
APPENDIX
CONTENTS
Analysis Details
18
A.1
Unimodal importance
18
A.2
Cross-modal interactions
18
A.3
Multimodal representations
19
A.4
Multimodal prediction
19
A.5
Sanity checks for saliency maps
19
B MULTIVIZ Visualization Tool
20
B.1
MULTIVIZ code framework
20
B.2
The MULTIVIZ website
21
B.3
Intermediate layer representation visualization
24
Datasets and Models
29
C.1
Datasets in MULTIVIZ
29
C.1.1
Multimodal fusion
................................
C.1.2
Multimodal retrieval
30
C.1.3
Multimodal question answering
30
Additional Experiments and Details
31
D.I
Model simulation
31
D.1.1
Setup
................................
D.1.2
VQA 2.0
32
D.1.3
MM-IMDb
32
D.1.4 CMU-MOSEI
36
D.2
Cross-modal interactions
38
D.2.1 CLEVR
38
D.2.2 FLICKR-30K
39
D.3
Representation interpretation
40
D.3.1 VQA 2.0 D.3.2
40
CLEVR
43
D.3.3
MM-IMDB
43
D.4 Error analysis
43
D.4.1
Setup
43
D.4.2
VQA 2.0
44
D.4.3
CLEVR
45
D.5
Model debugging
46
D.5.1 VQA 2.0
46
D.6
Summary takeaway messages
47
Limitations and Future Directions
48
E.1
Limitations
48
E.2
New classes of interpretation tools
49
E.2.1
Cross-modal interactions
49
E.2.2
Multimodal prediction
E.3
Evaluating interpretability
49
E,4
Engagement with real-world stakcholders
49
17

---

Published as a conference paper at ICLR 2023
A ANALYSIS DETAILS
A.1 UNIMODAL IMPORTANCE
Unimodal importance aims to understand the contributions of each modality towards modeling and prediction. It builds upon ideas of gradient-baset visualizations (e.g., Gradient Simonyan et al. (2013); Baehrens et al. (2010); Erhan et al. (2009)) and feature attributions (e.g., LIME (Han et al., 2020; Ribeiro et al., 2016; Yosinski et al., 2015), Shapley values (Merrick and Taly, 2020; Rodríguez-Pérez and Bajorath, 2020; Sundararajan and Najmi, 2020)).
Taking LIME (Ribeiro et al.. 2016) for an example, given model f. we would like to return weights over each of the 21 and z2 's such that important modalities are accurately weighted. LIME perturbs the set of Iı and Z2 's observes how model predictions change, and fits local linear model with re- spect to that datapoint. The areas with the highest positive weights are presented as the important ones. Other feature attribution and visualization approaches, such as Gradient based (Chandrasekaran et al., 2018; Selvaraju et al., 2017) or Shapley values (Merrick and Taly, 2020; Rodríguez-Péne Pérez and Bajorath, 2020; Sundararajan and Najmi, 2020), work similarly (Yosinski et al., 2015).
We implement unimodal feature attribution methods as a module UNI(fe ff. y, x) taking in a trained model fa. an output/feature which analysis is performed with respect to, and the modality of interest x. UNI returns importance weights across atoms of modality x.
A.2 CROSS-MODAN INTERACTIONS
Cross-modal interactions describe ways in which atoms from different modalities can relate with each other and the types of new information possibly discovered as result of these relationships. MULTIVIZ includes two recent methods for understanding mooss interactions:
EMAP (Hessel and Lee, 2020) decomposes f x22 = gi (21) + 92(22) + 912(21, 12) into strictly
unimodal representations 91,92, 92. and cross-modal representation 912 f - E,,, (f) E,g (f) E to quantify the degree of global (across an entire dataset) cross-modal interactions captured by a model.
DIME (Lyu et al., 2022) further extends EMAP by designing an efficient method for feature visual- ization on each disentangled representation locally (per datapoint).
Higher-order Gradient is our proposed method for efficiently quantifying the presence of cross- modal interactions. Based on the gradient definition of statistical non-additive interaction (Friedman and Popescu, 2008; Tsang et al., 2019), a function f exhibits non-additive interactions among
unimodal atoms I1 and x2 if
[a f(a).=))
> O. Writing the multimodal model f as f( x22 =
dwidez
(21) + (2() + (12( we can isolate the effect of 912(11, by taking a second-order
gradient of f with respect I1 and 22 the 9i (x1) and 92(22) terms becomes zero. Theoretically, second-order gradients are necessary and sufficient to recover cross-modal interactions: purely
additive models will have strictly 0 second-order gradient information, and any non-linear interaction term 012 (21, x222 must have strictly non-zero second-ords -order gradient information.
Definition 2 inspires us to extend first-order gradient and perturbation based approaches (Han et al., 2020; Ribeiro et al., 2016; Yosinski et al., 2015) to the second order. Our implementation first computes a gradient of f with respect to one input modality atom (e.g., I = birds), which results in a vector V Øf the vector components of V, via summation to produce single scalar ||V before taking second- order gradient with respect to all atoms of the second modality 322 E X2 (e.g., all image pixels), which
of the same dimension as Ti (i.e., token embedding dimension). We aggregate
dz,
results in vector V12- =
82f
82
of the same dimension as |x| (i.c., total number
Ba
of pixels). Each scalar entry in V highlights atoms I2 that have non-linear interactions with the original atom Fi (e.g., only the birds in the image, see Figure 2 for examples on real datasets). We implement a general module CM(fe, y,I,,xy x2) for cross-modal visualizations, taking in trained model fe. an output/feature y. the first modality's 's atom of interest 21, and the entire second modality of interest x2. CM returns importance weights across atoms 22 of modality x2, and can build on top of any first-order unimodal attribution method (i.c., gradient visualization (Erhan et al., 2009), LIME (Ribeiro et al., 2016), or Shapley values (Merrick and Taly, 2020), see Appendix A.2).
18

---

Published as a conference paper at ICLR 2023
We plan to make several approximations: only estimating single instances (æ1 222) at time which [f(=) as a measure of cross-modal interaction strength. Specifically, given model f. we first take a gradient of ʃ with respect to an input word (e.g., #1 dog), before taking a second-order gradient with respect to all input image pixels X2. which should result in only the dog in the image being highlighted (see Figure 2 for examples on real datasets).
avoids the expectation, and computing the magnitude w(F1, 22) = 2iy
We implement a general module CM(fo, y. 21, x1,X2) for cross-modal visualizations, taking in a trained model fo, an output/feature y, the first modality's atom of interest æT, and the entire second modality of interest X2. CM returns importance weights across atoms 2/2 of modality X2, and can build on top of any first-order unimodal attribution method, such as gradient visualization (Goyal al., 2016), LIME (Ribeiro et al., 2016), or Shapley values (Merrick and Taly, 2020).
A.3 MULTIMODAL. REPRESENTATIONS
Given these highlighted unimodal and cross-modal interactions at the input level, the next stage aims to understand how these interactions are represented at the feature representation level. Specifically, given a trained multimodal model f. define the matrix M, R wxd as the penultimate layer of representing (uninterpretable) deep feature representations implicitly containing information from both unimodal and cross- modal interactions. For the ith datapoint, 2 (t) collects a set of individual feature representations 21, 22, .... Zd E R. We aim to interpret these feature representations through both local and global analysis (see Figure for an example):
Local representation analysis (R₂) informs the user on parts of the original datapoint that activate feature 2j. To do so, we run unimodal and cross-modal visualization methods with respect to feature 2j (i.e., UNI( 2j, x). CM 2,. 21 X2)) in order to explain the input unimodal and mrosl interactions represented in feature zj. Local analysis is useful in explaining model predictions on the original datapoint by studying the input regions activating feature Zj.
Global representation analysis (Rg) provides the user with the top datapoints Dz(2z) {(x1, X2, (1)) } that also maximally activate feature 2j. By further unimodal and mroal visualizations on datapoints in (zjz)), global analysis is especially useful in helping humans assign interpretable language concepts to each feature by looking at similarly activated input regions across datapoints (e.g., the concept of color in Figure 1). Global analysis can also help to find related datapoints the model also struggles with for error analysis.
A.4 MULTIMODAL PREDICTION
Finally, the prediction step takes the set of feature representations 22, ..., 2d and composes them to form higher-level abstract concepts suitable for a task. We approximate the prediction process with linear combination of penultimate layer features by integrating a sparse linear prediction model with neural network features (Wong al., 2021). Given the penultimate layer M, Nxd, we fit linear
model E (Y|X = x)) = .M. 3 (bias 30 omitted for simplicity) and solve for sparsity using:
1 2N
β = arg min
J B = 1111 + 2211
The resulting understanding starts from the set of learned weights with the highest non non-zero coef-
ficients Rto = {.(1), ...) and corresponding ranked features zoop = {2(1)- 2(2)...... Stop tells the user how features zoo are composed to make a prediction, and zoop can then be visualized with respect unimodal and cross-modal interactions using the representation stage.
A.5 SANITY CHECKS FOR SALIENCY MAPS
According to Adebayo et al. (2018), a vistualizatio//ineerrrttttoo method should be rejected if admits invariance over either data model, i.e. transformation of data model does not change the output of the method. We perform a similar sanity check on MULTTIVI:
Data randomization test: MULTIVIZ does not admit data invariance, as MultiViz visualizations on the same model varies between different data points and labels (see visualization examples in Figure 4, 5, and 6). The visualizations reliably capture unique input regions, related datapoints, feature concepts, and errors specific to each data point.
19

---

Published as a conference paper at ICLR 2023
Table 4: We scaffold the problem of interpreting multimodal models into the following stages for which algorithm design and analysis can occur. For each step, MULTIVIZ includes existing and newly proposed approaches for visualizing models across modalities and tasks.
Level
Methods
Gradient (Simonyan al., 2013; Bachrens al., 2010; Erhan al., 2009), LIME (Han et al., 2020; Ribeiro et al., 2016; Yosinski et al., 2015).
Unimodal importance
SHAP (Merrick and Taly, 2020; Rodríguez- Pérezz and Bajorath, 2020)
Cross-modal [ (Gradient, LIME, SHAP) (new).
Cross-modal interactions
EMAP (Hessel and Lee, 2020), DIME (Lyu et al., 2022)
Multimodal representation
Local global analysis (new)
Multimodal prediction
Sparse linear model (new)
Algorithm I Visualizing and understanding multimodal models using MULTIVIZ.
Given:
Dataset D = {(x1, X2, y) } ((x) ..... 22 ........................ y) } and trained model fe.
Given: Unimodal and cross-modal visualization subroutines UNI (NN(So, v... x), CM(So, V. 21, X2). Obtain deep features M, = fo X2) for datapoint of interest.
Fit sparse linear model: fesparse = M 3 by solving equation (2).
Obtain predictions ; = [Asparse (X) x2) and ranked features zup with largest coefficients.
Visualize overall unimodal importance wrt ÿ: U1 UNI(JO, D. x1), U2 = UNI(Jo, U, x2). Visualize overall cross-modal interactions wrt ÿ: C = CM, 9. 21, X2). CM G, 22, x1). for each top feature in Stop do
local ana analysis for original datapoint Visualize unimodal and cross-modal interactions of original datapoint wrt global analysis across similar datapoints
Obtain top k datapoints that also maximally activate z. for each new top k datapoint (x1, X2, y) do
Visualize unimodal and cross-modal interactions of new datapoint wrt
Model randomization test: In Appendix D.2, we demonstrated that MULTIVIZ produces different results for two different models on the same data for both CLEVR question answering and Flickr-3KK retrieval: MULTIVIZ enables to explain differences in performance across 2 models based on the accuracy of cross modal interactions each model captures, so MULTIVIZ passes the model randomization test.
Therefore, methods do not admit data or model invariance and passes the sanity checks from Ade- bayo et al. (2018).
B MULTIVIZ VISUALIZATION ToOL
We summarize these proposed approaches for understanding each step of the multimodal process in Table 4. and show the overall pipeline in Algorithm and Figure To enable human studies. MULTIVIZ provides an interactive API where users can choose multimodal datasets and models and be presented with a set of visualizations at each stage.
In this section, we will include both introductions our code framework that enables easy application of analysis visualization methods to datasets and models, and also present the MULTIVIZ iz website that showcases some examples of visualizations generated for each stage on different datasets and models.
B. MULTIVIZ CODE FRAMEWORK
One additional major contribution our works is that we designed a code framework in Python for easy analysis, interpretation and visualization of models multimodal datasets with only few lines of code. The framework is modularized and extendable to new datasets, models and visualization methods. Figure 7 is an illustration of the main modules of the code framework:
Within the datasets module, include scripts for retrieving information directly from the dataset, including getting specific data points from a split, getting the ground truth labels, label-id-to-answer and answer-to-label-id mappings, etc. Some dataset scripts also
20

---

Published as a conference paper at ICLR 2023
Datasets
Models
VQA 2.0
CLEVR
CMU-MOSEI
LXMERT
ViLT
CLIP
MIMIC
Flickr-30k
MM-IMDb
CNN-LSTM-SA
MulT
LRTF
Analysis methods
Visualization tools
LIME
FoG
SoG
LIME
Gradient visualizer
Sparse
visualizer
linear
EMAP
DIME
Sparse
model
linear
visualizer
model
An illustration of the modules available in code framework. Each dataset class provides data loading and label-answer mapping for a particular multimodal dataset; each model class is a wrapper for particular model a dataset and supports functionalities like making prediction and taking gradients; each ana lysis script performs a certain analysis method (such LIME) on arbitrary input data point and model wrapper; and the visualization scripts are tools to visualize analysis results.
Figure 7:
supports generating visualizations for data points (for example, the script for VQA supports generating pictures that contain both the image and the question).
Within the models module, we write a wrapper for every supported model that inherits a common parent class called analys smodel, which defines set of functionalities commonly used in various analysis methods. The functions analysi mmalysismodel smodel in- clude forward (just making prediction (forward but on multiple points a batch), getgrad (compute gradient, applicable), getpre inear (getting representation features), and many others. This design allows the same analysis script to work on vastly different models, as long as the models are wrapped by class that shares these functionalities.
specific data point), Forwardbate ch
Within the ana 1 yss s module, we have scripts that can take in arbitrary data point and model class (that inherits analysi imaadesssmoell smodel) and perform various analysis methods such LIME, DIME, EMAP, Sparse Linear Model, etc. These scripts generate the outputs in numerical format without visualizations, and users can choose to visualize them arbitrary ways.
Within the vi visull 1 i zati ons module, we have scripts that provide tools to visualize the analysis results from the analysis module.
In Algorithm we showcase an example of running LIME, DIME, Sparse Linear Model and represen- tation feature analysis (local and global), thus covering all stages of MULTIVIZ. As you can see, the code is actually very short (without the comments) for running this many analysis and visualizations. Our code framework is also easily extendible to support new datasets, models and analysis/ visualiza- tion methods, by writing and adding scripts to the datasets/moddel//aallls visiaaiiiiiiissuliaatttnnn modules respectively.
B.2 THE MULTIVIZ WEBSITE
We also created visualization website accompanying MULTIVIZ which organizes visualizations of all stages on a particular datapoint of specific dataset-model pairs. The URL link of the webpage is available at https: /tttpp://gittuii //gptttttttttt//////////tt www.pooaanaee..cccc (aapg/2799/91111 /iizz
Figure 8 is one example webpage for data point in VQA. On the left there is a control panel that allows users to switch between different datasets and instances (i.e., data points), and then below the two boxes shows all information about the data point (image and question in the case of VQA) and also the ground truth ("GT") label and the predicted ("Pred") label. On the right side, we have a graph showing simplified version of the Sparse Linear Model: we only show the top 5 features with the
21

---

Published as a conference paper at ICLR 2023
Algorithm 2 Example of generating visualizations for VQA using our code framework.
from datasets. vqa import VOADataset A import the dataset from models. vga. lxmert import VOALXMERT $ import the model import analysis methods
from analysis. unimodallime import rununimodallime
from analysis dime import dime
analysis.. mmalyyLa.Spazeet.nezzre sparset..earzncooding import get_ prt_sparre._ linear model i import visualization tools
from visuslizations visualizelime import visualizelime
visualizations... www.ooowwwaaeeeeerriaaaaaaaoiiio inneaarrraelllammddall.... import amllyrapchtanddkkmmlleeellll maalyyefettureaanttssaaaaeeeee
sparsityacograph
get model, and predictions datas vgADataset ('val') analysismodel (CALXMERT('cun ('cuda:0')
instance datas. getasage predlabel amalysismodel. www.eeedeeggepeeebbbeeeeeeeeeele (snalysmmmmme. wow.:modee.oooodddiieeeeeeeoeeee (istard() correctlabel
getcorrctatto (instance)
run and visualize unimodal importance on predicted label explanationl rununimodallime (instance, 'image', 'image' analysismodel, (predlabel]) visualizelime (explaantti 'image' .predlabel, ) explenation2 rununimodall (instance, 'text', , 'text' ,aaaleiimodel,, (predlabel)) expl
visuslizelime (explanation2, 'text' .prddaaell, "imagelime.png"
run and visualize cross-modal interactions on predicted label instanceset = (ieiaa..getdataai45505
for i in (100)] explanations dime (instanceset, anl,, amalyyismodel, [predlabel]) visualizelime (explamatiiiilaiiioooo [0], 'image' .0, ismagedineumimmcall...o
visualizelime (explanations [0], 'image' ' ,1, manggedimmuuttmmdal..... png')
visualizelime (explanation) ions [1], 'text', "textdimeunimddll png'
visualizelime (explanations [1], 'text' ,,, "textdimemultimodan"
train linear model and visualize
params, get sparse linear-model (amalysismodel, 'trainfeats.pue 'valfeats, pkl'', 'valfeats.pkl" pparsttyacegraphhree. 'sparseplot.png
run sampledata datas datas. getseqdata 20000)
global analysis
local analysis
mellyzepointenndvieaaal..ez (params, instance, anw.yylisooell. ptt.://www.......aaaaaaeeeeeeeee 'tmp/local', 'local')
global analysis
maalyeefaturresnnviisuulleelll (params, instance, sampledata, analysismodel, preeleii predlabel, 'tap/global', 'global')
MultiViz Simulation Experiment
Delieat:
Tratance:
uaiC mnsiin Lagle dadds with opt coaprtedio
Unandd Dan itags
2010
Figure 8: An example of MUL.TIVIZ webpage for VQA (Overview page). Best viewed zoomed in and in color.
highest weights for each label (the weights are shown as numbers on the lines). Note that we will show both correct and predicted labels in the graph (so if the model got the answer wrong, there will be two labels shown under "classes" as shown in Figure 10, and clicking on each label will navigate to webpage that shows visualizations with respect to that specific label). In the middle tab titled Main View, we show the visualizations from U and C stages. In the case of VQA we present unimodal LIME U stage visualization (first column under Ma in View) and DIME C stage visualization
22

---

Published as a conference paper at ICLR 2023
MultiViz Simulation Experiment
Doneer
Instane::
Fliwert Replenation
wocodt the
Tower ELM
Tarver Tee
Raclourd Soplenation
Eatt of
Dedivied Stal Caang
Dacowed 3.M Danpkd
Eedroved Canppin lagg
Figure 9: An example of MULTIVIZ webpage for VQA (Features page). Best viewed zoomed in and in color.
MultiViz Simulation Experiment
VIS
Ceeninw of chen 11 bim areeyal
Lieee Ban cntti
srvame ratir witeral the
Viaa VOAVAl inilnce 205
caee plolic
Unvecaal Lemolrncc
Uninde
Matrodi Din
Mus 411)
orngeoo 161)
Figure 10: An example of MULTIVIZ webpage for VQA (Overview page). Best viewed zoomed in and in color.
(second and third column under Main View). We call this webpage the Overview webpage. For each of the top five representation features shown within the graph, the user can access R, and R, visualizations of each feature by clicking the circle in the graph representing that feature and the user will see a feature webpage like Figure 9. Under Main View, we include local analysis visualizations (unimodal lime with respect the feature in the of VQA) the top and then global analysis visualizations on the bottom. To return to the Overview page, the user can just press the label cirele under "classes" in the graph on the right again.
We also show additional example webpages: MM-IMDb (Figure 11 and Figure 12, with first order gradient for U stage, second order gradient for C stage), CMU-MOSHI (Figure 13 and Figure 14,
23

---

Published as a conference paper at ICLR 2023
MultiViz Simulation Experiment
Donaae:
Iratano::
1256
beady witt oight
rage snatec dnady oggir
cran entarting
umvoctillyy
Zarpolinecs sical
arrouiis trring le orew abol the sann ajuct torn cuanass thrr cotum recrannutee Zarool
samprro seac umich cornciare vim Deserntion intingitort
Thanderbrss eeronac
redeugged mede, adlmmnn tiny.
sowe Joce Treey degii Trocy, llee Trncy and deetts Tincy
Thanclrtic coe
pereegg
creg stnvicars the inuneriade out snrynocee to prover pouate czadtage a secord the arouns reqires
beact Fanaloge
Uniacce Coder Red
Thoough the T'uade boak eussio, an tegins soprodaaadd na los, which camenase Zarai.
BD
Figure 11:
An example of MULTIVIZ webpage for MM-IMDB (Overview page). Best viewed zoomed in and in color.
with first order gradient for U stage, second order gradient for C stage) and MIMIC (Figure 15, with first order gradient for U stage). Note that we only ran stage for MIMIC LF model because its cross-modal interactions
negligible (second order gradients are all zero) and there too few representation features to do sparse linear models.
We have also used modified versions of these webpages conduct all experiments with human annotators. See Appendix D for details.
B.3 INTERMEDIATE LAYER REPRESENTATION VISUALIZATION
Our codebase is designed such that the user may specify any layer in a model as the representation and run R, and R, analysis neurons in that layer. We showcase few examples of R and R, on neurons on the third-last layer on LXMERT model on the VQA dataset in Figure 16.
The reason choose the P stage as will just be linear composition. we use different layer as our representation, the stage will contain multiple layers with different weights and more complex interactions, making more difficult for a human user to visualize how each neuron in the representation related to the final prediction.
the second last layer in the models is mostly for ease of visualization in
24

---

Published as a conference paper at ICLR 2023
MultiViz Simulation Experiment
Deliont::
Fostrr 10666F)
Tratane::
1220
Fonwerd Explenatio
ddticccal
the suec
hnd deroot codurtttyy
drout
faoe list sbor ctfou lexs forshgg trew sduet
litc, redinagnnt are recesoruceed deou many for pecr commediel chlingg mush Man,
Expieaatoon
cannrage Accurn Bayy the hutee bolki. eneetmreu Jul Thecranche bou sore Soot Sonder
douits: akce mucroo
bash be souuduul Lubb threage decplics Thandcott proeect
aed make
me homer mnme Coore fven free
mn poede freer dengett
fimbo boat nonur uith
Socalin bac dccient sos thi
of fee aiti dbll
en tose sevieee ovny pore
cavern unrtiry gees uvee
resoeer perire duem paen
Deckoe burgki
Eeckuaed Campald laage
Bodkaad Eoumuie
brardk
Batwaed 2 Mibuerl
Figure 12: An example of MULTIVIZ webpage for MM-IMDb (Features page). Best viewed zoomed in and in color.
MultiViz Simulation Experiment
Gedcnkbent maseeeeee
"Mlet Teat""
ho" "woltiegg het 'V"
calence branee and bony Doss wend ie
Comon
harr caher set har
Iegalis Sc
Vae cimrsoau sucant
Jeetiv 2019
Secar Gtrdi #4620
Undodds GeadertAdee
Figure 13: An example of MULTIVIZ webpage for CMU-MOSEI (Overview page). Best viewed zoomed in and in color.
25

---

Published as a conference paper at ICLR 2023
MultVvzz Simulation Experiment
Conaci
Ootbaa:
Paoken di agele rbox Drood
Viee
ICCEI 20070
Feseecum Jamin "he move hase
Towey
Colttore one na wite junney and ans sten Johns
Legi to
1928
segaive 0e
Fowere
Favverd SLMiddd
seatve
Bacwardd Expiomation
Deckoeet Exandee
Endroard SLM Camprii
Butonen Orain
Bednaadd Sut Trahie Tratrrhaahss hrahos
E Weve Toaker Aratoo
Tedivaed SUM Changiat
Dariveed SL M'Steppdd Tex
Figure 14: An example of MULTIVIZ webpage for CMU-MOSEI (Features page). Best viewed zoomed in and color.
26

---

Published as a conference paper at ICLR 2023
MultiViz Simulation Experiment
Marvee
Deliaat:
wordi
MMMECOO0T metorme 13
Cosi
Onle Stend
Orae Oradend Sutvel
(010)
diceoo 3019)
Figure 15: An example of MULTIVIZ webpage for MIMIC (Overview page). Best viewed zoomed in and in color.
27

---

Published as a conference paper at ICLR 2023
18396
18699
12851
100
200
100
VISIIAB
200
300
3.0
300
200
200
600
500
100
100
300
500
100
VISI
200
100
300
300
200
400
500
300
500
Una eptenieele we (ieo
Uncal ney audien (leck
-00
100
200
200
0700
300
400
-23
-02
420
-22
-20
-1.0
-101
-44
700
500
300
6753
6180
5119
wock
Quedler Vil udtigeate
200
300
200
400
300
400
100
200
300
400
300
600
100
200
300
400
500
600
100
200
:00
400
500
600
100
200
200
200
400
400
400
100
200
300
400
500
600
100
200
300
400
500
500
100
200
300
400
500
600
300
100
012
700
300
300
300
300
400
600
120
300
800
500
Figure 16: Examples of running R, and R, analysis third-last layer neurons of LXMERT VQA dataset. On the top image (Neuron of the layer), clearly this neuron represents sports-field related image; and on the bottom image (Neuron 100 of the layer), clearly this neuron represents "lying/sitting on" relationship in the question.
28

---

Published as a conference paper at ICLR 2023
C DATASETS AND MODELS
All of our datasets build upon diverse and standardized set of multimodal benchmarks in Multi- Bench (Liang et al., 2021a). We briefly describe the datasets and preprocessing here:
C.I DATASETS IN MULTIVIZ
C.1.1 MULTIMODAL FUSION
In multimodal fusion, the main challenge is to join information from two or more modalities to perform a prediction. Classic examples include audio-visual speech recognition, where visual lip motion is fused with speech signals to predict spoken words (Dupont and Luettin, 2000). Information coming from different modalities have varying predictive power by themselves and also when complemented by each other (i.e., higher-order interactions). interactions, there is also need to identify the relations between granular units from two more different modalities (i.e., alignment). When dealing with temporal data, also requires capturing possible long- -ragge dependencies across time (i.e., temporal alignment). MULTIVIZ contains the following datasets for multimodal fusion spanning:
order to capture higher-order
(1) CMU-MOSEI is the largest dataset of sentence-level sentiment analysis and emotion recognition in real-world online videos (Liang et al., 2018 8a; Zadeh et al., 2018) with more than 65 hours of annotated video from more than 1, 000 speakers and 250 topics. Each video is annotated for sentiment as well as the presence of 9 discrete emotions (angry, excited, fear, sad, surprised, frustrated, happy, disappointed. and neutral) as well as continuous emotions (valence, arousal, and dominance). The valuable dataset to test multimodal models
diversity of prediction tasks makes across a range of real-world affective computing tasks. The dataset has been continuously used in workshops and competitions revolving around human multimodal language.
Dataset preprocessing: We follow current work (Liang et al., 20188 8b; Zadeh et al., 2018) and apply standard preliminary feature extraction for the CMU-MOSEI dataset.
Train, validation, and test splits: Each dataset contains several videos, and each video is further split into short segments (roughly 10 20 seconds) that are annotated. We split the data at the level of videos so that segments from the same video will not appear across train, valid, and test splits. This enables us to train user- indepenndnndt models instead of having a model potentially memorizing the average affective state of user. There are a total of 16, 265, 1, 869, and 4, 643 segments in train, valid, and test datasets respectively for total of 22, 777 data points.
(2) MM-IMDB is the largest publicly available multimodal dataset for genre prediction on movies (Arevalo et al., 2017). MM-IMDB starts from the movies of the MovieLens 20M dataset and expands this dataset by collecting genre, poster, and plot information for each movie. The final dataset contains ratings for 25, 959 movies. MM-IMDB realistic real-world multimodal dataset and is popular benchmark for multimodal learning (Arevalo et al., 2017; Kiela et al., 2019; Pérez-Ráa et al., 2019).
Dataset preprocessing: We used the same method as (Arevalo al., 2017) to extract features from texts and images.
Train, validation, and test splits: The MM-IMDb dataset is split by genre into train, valid, and test datasets containing 15552, 2608, and 7799. The split was performed so that training, valid and test sets comprise 60%, 10%, 30% samples of each genre respectively.
(3) MIMIC-III (Medical Information Mart for Intensive Care III) (Johnson al., 2016) a large, freely- database comprising de-identified health-related data associated with over 40, 000 patients who stayed in critical care units of the Beth Israel Deaconess Medical Center between 2001 and 2012. Following (Purushotham et al., 2018), we organized numerous patient data into two major modalities (using the features in feature set A in (Purushotham et al., 2018)): time series modality, which is a set of medical measurements of the patient taken every 1 hour in a period of 24 hours. Each measurement is a vector of size 12 (12 different measured numerical values); static modality, which is set of medical information about the patient, represented in vector of size 5. We use these modalities for 3 tasks: mortality prediction (6-class prediction on whether the patient dies in 1 day, 2 day, 3 day, week, 1 year, or longer than 1 year), and 2 ICD-9 code predictions (binary classification on whether the patient fits any ICD-9 code in group 1 (140 - 239) and binary classification on whether the patient fits any ICD-9 code in group 7 460 519). MIMIC poses unique challenges in integrating
29

---

Published as a conference paper at ICLR 2023
time- varying and static modalities, reinforcing the need of aligning multimodal information at correct granularity.
Dataset preprocessing: We followed the instructions on https: http://mmmmm....miiim.omm
acceesss//ccc/////eee/ooss to download the dataset in the form of raw tables, then generated preprocessed data following the steps described in https:/ https://github. com/UsC-Melady baddy//eecmmariigggg........aa.. BB______MMMIIIII time) get the data used for experiments. Specifically, we will use data in the file 24hrs/ wthrs/series seriesrr////// imppimm nortee---rrme-eeee ep__ ep_1__2_1420221 1e____2-stdiieeeeeeeeeeeeeeeeeae stdized. npz. When accessing this data from our code repo, set the imputed happputed__phhhhhhhee of the npz file above in the script will generate the Py PyTorch data loader for the tasks (where we will normalize the data).
(which takes 1 - weeks running
data. py and the
Train, validation, and test splits: We split the data into train/valid/test sets randomly (using fixed random seed) in a 80 : 10: 10 ratio (so 28, 970 train, 3, 621 valid, and 3, 621 test data points) for total of 36, 212 data points.
C.1.2 MULTIMODAL RETRIEVAL
Another area of great interest lies in cross-modal retrieval (Liang et al., 2021b; Zhen al., 2019), where the goal is to retrieve semantically similar data from a new modality using a modality as query (e.g., given a phrase, retrieve the closest image describing that phrase). The core challenge to perform alignment of representations across both modalities. MULTIVIZ contains the following datasets for multimodal retrieval and grounding:
(1) FLICKR-30K (Plummer et al., 2015) contains 32, 000 images collected from Flickr, together with 5 reference sentences provided by human annotators enabling the tasks text-to-image ref- erence resolution, localizing textual entity mentions in image, and bidirectional image-caption retrieval.
Train, validation, and test splits: The training items generated from the captions of 25, 000 images, and the test items are generated from a disjoint set of 3, 000 images.
C.1.3 MULTIMODAL QUESTION ANSWERING
Within the domain of language and vision, there has been growing interest in language- baaed question answering (i.e., "query" modality) of entities in the visual, video, embodied domain (i.e., "queried" modality). Datasets such as Visual Question Answering (Agrawal et al., 2017), Social IQ (Zadeh al., 2019), and Embodied Question Answering (Das et al., 2018) have been proposed benchmark the performance of multimodal models in these settings. A core challenge lies in aligning words asked in the question with entities the queried modalities, which typically take the form of visual entities in images or videos (i.e., alignment). MULTIVIZ contains the following datasets for multimodal question answering spanning several research areas:
(1) CLEVR (Johnson et al., 2017) is diagnostic dataset for studying the ability of VQA systems to perform visual reasoning. contains 100, 000 rendered images and about 853, 000 unique automatically generated questions that test visual reasoning abilities such as counting, comparing, logical reasoning, and storing information memory.
Train, validation, and test splits: The complete dataset contains more than 608K train, 140K val and 140K test (question, image) pairs.
(2) VQA 2.0 (Goyal et al., 2017) is a balanced version of the popular VQA (Agrawal et al., 2017) 7) dataset by collecting complementary images such that every question is associated with not just a single image, but rather pair of similar images that result in two different answers to the question. The reduces the occurrence of spurious correlations the dataset and enables training of more robust models.
Train, validation, and test splits: The complete balanced dataset contains more than 443K train, 214K val, and 453K test (question, image) pairs.
30

---

Published as a conference paper at ICLR 2023
Unimodal Importance
y
Cross-modal
Interactions
Multimodal
representations
31
32
Multimodal
prediction
human studies determine if users are able to simulate model predictions given only MULTIVIZ visualizations. If MULTIVIZ indeed generates human- understandable explanations, humans should be able simulate model behavior accurately for both correct and incorrect model outputs.
Figure 17: Model simulation:
D ADDITIONAL EXPERIMENTS AND DETAILS
In this section, we provide additional details on the experiments and additional results on several other multimodal datasets.
Computational resources: Preparations for all experiments (i.e. generating the necessary visualiza- tions for the points for each dataset) are done on a private server with 2 GPUs.
The preparation time for model simulation experiment using GPUs is about 12 hours for VQA, 1 hour for MM-IMDb and 2 hours for CMU-MOSEL For the representation interpretation experiment, generated all visualizations for the VQA data points in the experiment in about 3 hours on 1 GPU. For the error analysis, in addition to the visualizations already present on the MultiViz website, we also have GPU available live during the human annotation (so human annotators can request second order gradient analysis on specific words, each second order gradient computation only takes 2-3 seconds).
In all of the above analysis, we re-use the sparse linear model we had already trained for each dataset when building the main MultiViz webpage (the initial training can take some time scaling the Sparse Linear Model to the large VQA took over 72 hours with 1 GPU).
Note that VQA visualization generation is much slower than those for MM-IMDb and CMU-MOSEL. This is because in VQA used DIME for cross-modal interaction interpretation, but in MM-IMDb and CMU-MOSEI we use second-order gradient. The newly proposed second order gradient is much faster compared DIME since it only requires running the model once instead of up to 10,000 times in DIME.
Overall, the proposed MULTIVIZ interpretation stages are efficient and only add negligible time on top of existing trained models, especially for our newly-proposed second-order gradient method.
Participant risks and compensation: and without compensation. There Participation in these human studies were fully voluntary participant risks involved. We obtained consent from all participants prior to each short study. All annotations are fully anonymous and we do not store any information regarding the participants at all.
D.1 MODEL SIMULATION
D.1.1 SETUP
We design a large-scale use case of model simulation to determine if MULTIVIZ helps users of multimodal models gain deeper understanding of model behavior, shown in Figure 17. We design a human study to see what humans predict given MULTIVIZ explanations at each step (and across all steps). If MULTIVIZ indeed generates human- understandable explanations, humans should be able to make a prediction on the task given these explanations only. Specifically, we compare the full version of MULTIViZ with set of local ablations, each consisting of only additional stage:
1. U: Users are only shown the unimodal importance (U) of each modality towards the prediction. + C: Users are shown both unimodal importance (U) and cross- modsl interactions (C) high- lighted towards the final prediction.
+ Rg: Users are shown unimodal importance (U) and cross-modaa interactions (C) of the given datapoint highlighted towards the final prediction, well as local analysis (Rş) of unimodal
31

---

Published as a conference paper at ICLR 2023
and cross- modsl interactions of top ranked feature representations zwp ((2)).)()...))) with respect to that local datapoint.
4. U + C+ R2 Rg: Users are additionally shown global analysis (R₂) through similar datapoints that also maximally activate those same top ranked feature representations.
5. MULTTIVIZ (U + C + R2 Rg + P): This constitutes the entire MULTIVIZ framework by including visualizations of the final prediction (P) stage: sorting all top ranked feature neurons
2top = {z( =(2),...} with respect to their coefficients Srop (12)))(2))->> (..) and showing these coefficients to the user.
We ask human annotators (who all have or are currently working towards a B.S. in a STEM field and have at least basic knowledge of machine learning models) to predict the output of a model analysis results and visualizations. In each of the following datasets (VQA 2.0, MM-IMDb, CMU-MOSED), we divide 15 total human annotators into 5 groups of 3, each group getting one of the five settings above, and then we compute average accuracy and inter-rater agreement within each group. The full results are shown in Table 2.
D.1.2 VQA 2.0
In this experiment, we will perform model simulation on VQA 2.0 dataset with pretrained LXMERT (https: http://huggingg htthhhhgggggaacceeeee/o/ co/ure uuwunlccccpp//////e tmeert-vqa- unaased). . We randomly selected
22 points from the validation split of the VQA dataset under the following criterion: (1) it is not yes/no question and (2) the answer to the question is not infrequent (i.e. it occurs at least 220 times over 220K+ validation points). For each of the point, for U stage we run LIME on each modality; for C stage we run DIME; for R we run LIME with respect to the representation feature this data point; and for R2 we run LIME each modality with respect to the representation feature on 3 examples that maximally activates the feature; and for P we show the top 5 representation features with the highest weights with respect to the predicted class in a Sparse Linear Model trained on the training set of VQA. The webpage for each datapoint is organized into Overview page (containing U and C) as well as five Features page (R, and R, for each of the top 5 representation features) as well as a "graph" on the right showing P. An example Overvier page is shown in Figure 18 and an example Features page is shown in Figure 19. In settings (1)-(4), we will use versions of the webpage with certain stages removed (for example, Figure 20 is the webpage for setting (2), only showing U and C).
MULTIV IZ analysis and visualization:
Within each of the five groups, on each of the 22 points, human annotators are asked to predict what the model (LXMERT) predicts given a website containing some or all of the stages of analysis visualizations (depending
the group's s setting). In addition, they are given an answer sheet (see Figure 21) where they are given 4 answer choices for each data point to predict with, and they have to select one of the choices they think LXMERT most likely predicted as the answer to each data point. Before each annotator starts, they are taught how to interpret each analysis visualization, and then the instructor goes over points together with the annotators as examples and the annotators need to finish the remaining 20 points on their own. Only the remaining 20 points counts towards the data collected in the experiment. We then compute average accuracy and inter-rater agreement score (Krippendorff's s alpha) within each group. In addition, groups under settings (3), (4) and (5) are asked whether they found the Overview or Features page more helpful.
As shown in Table 2, in general, human annotators were able to better predict the model's 's predictions when they were given more information, as the groups that got more information almost always end up with both higher average accuracy and higher inter-rater agreement. Moreover, annotators in settings (3), (4), (5) reported that they found Features page most helpful compared to Overvi i ew page 31.7%, 61.7% and 80.0% of the time respectively, therefore showing that R,, and P helps make representation analysis a lot more useful.
1.1.3 MM-IMDB
In this experiment, we perform model simulation on MM-IMDb dataset with the LRTF model from MultiBench (Liang et al., 2021a). We randomly selected 21 points from the test split of MM-IMDb dataset. The original MM-IMDb dataset is designed for multi-label classification, but for simplicity, we only take the label with the highest prediction probability from LRTF as the predicted class, and effectively treat it as a single-label classification task during analysis, visualization and model simulation experiment. For each of the points, we run MULTIVIZ analysis and visualization: for stage we show first order gradient analysis on image and text; for C stage we perform second order gradient analysis on the top ten words with maximum first order gradient; for R, we show
32

---

Published as a conference paper at ICLR 2023
Danase::
model
Pictanda:
DBE oetoo dectinping mood
ne diety or me orts dem
Qunrbe
Hon war lidk?
image
Unrovdu Orn Irout
tnage
Figure 18: Simulation experiment for VQA: MULTIVIZ website Overview page showing LIME and DIME explanations. Best viewed zoomed in and in color.
Contro
Sedkment Dxpare aten
opensor
Oriire Oriine
Figure 19: Simulation experiment for VQA: MULTIVIZ website specific representation feature
showing forwards and backwards analysis (a Features page). Best viewed zoomed in and in color.
33

---

Published as a conference paper at ICLR 2023
Control Panel
Main View
Uctosee:
Owerdiay prodiction analysii
DNIE ong
rood
Iratancw: vQ4255
Uninoce LVC explnnaioon image and sest ua shacty on the sagior oftee cass
ent unveooaaton teltun mutt notil vntercttoon, nun (M6 orageden vacrs ptoctionpio par
Nita Varwitt instance 203
Glabilon
How mony birds?
Urancebe Iio kronys
Millinrrsd Daan image
2 200:0011
Uninodal Dine
Figure 20: Simulation experiment for VQA: Setting webpage with only LIME and DIME explana- tions. Best viewed zoomed in and in color.
A
C
Answer
255
3
1205
white
orange
blue
green
2305
skis
snow
shoes
ground
2605
male
female
1905
3205
man
woman
toddler
baby
3255
3405
white
orange blue
green
705
old
young
2655
white
orange
blue
black
855
left
right
up
down
905
bus
car
SUV
ambulance
955
2
6
4155
1
2
3
4255
2
3
4755
white
orange
blue
green
5555
red
yellow
blue
green
5605
cat
dog
horse
cow
6205
3
6255
3
4455
2
4
7505
cat
dog
horse
Figure 21: Simulation experiment for VQA: Multiple choice answer sheet given to the annotators.
34

---

Published as a conference paper at ICLR 2023
MuibViz Simulation Experiment
COO walje
de sicees utecrnitt
IODMA pobe peach
Bow ching
Figure 22: Simulation experiment for MM-IMDb: Sample Overvi iew ew page. Best viewed zoomed in and in color.
Experiment
tetak he manka olaun Neuse hoo cedera
Sedtoard
and de
with cut corns butrido
You mork sewr decsen of 1om and und meterneeee cate ctling
Endawi3 22 Coeadd
Soree 2MU Gareld
Figure 23: Simulation experiment for MM-IMDb: Sample Features page. Best viewed zoomed in and in color.
35

---

Published as a conference paper at ICLR 2023
Experiment peprmmeent
MM00 -iacitrphhrep
Gommther Uniente find
Wit and cmooor
Typ
Uniesa Borden mage
Figure 24: Simulation experiment for CMU-MOSEL: Sample Overview page. Best viewed zoomed in and in color.
first order gradient image and text with respect each representation feature; for Rg. each representation feature we present 3 data points that maximally activates the feature, and also show first order gradient visualization for each; for P stage show the "graph" top 5 representation features from Sparse Linear Model analysis as well as their respective weights. The webpage organization is the same as the webpage for VQA with the Overview page (Figure 22) and Features pages (Figure 23).
the right that ranks the
Within each of the five groups, on each of the 21 points, human annotators are asked to predict what the model (LRTF) predicts given website containing some all of the stages of analysis visualizations (depending on the group's ' setting). In addition, we give human annotators 10 possible movie classes that the model could predict for these 21 points ("Drama/Romance" "Crime", "Sci-Fi", "Comedy", "Thriller", "Western", "Action", "War". "Documentary", "Horror"). Note that in reality, not mutually exclusive, but we intentionally designed our experiment
some of these categories this way to see if human annotators were able to determine the model's prediction by looking at what specific properties within the movie's 's poster or description the model focused on during the prediction process. Before each human annotator starts, they are taught how to interpret each analysis visualization, and then the instructor goes over the first point together with the annotator as example and the annotator need to finish the remaining 20 points their own. Only the remaining 20 points counts towards the data collected in the experiment. We then compute average accuracy and inter-rater agreement score (Krippendorff's 's alpha) within each group.
As shown in Table 2, in general, human annotators were able to better predict the model's predictions when they were given more information, the groups that got more information almost always end up with both higher average accuracy as well as higher inter-rater agreement. We were especially surprised to find that including stage actually helped, since MM-IMDb did not seem to be a task that relies much cross-modal interaction.
D.1.4 CMU-MOSEI
In this experiment, we perform model simulation on CMU-MOSEI dataset with the MulT model from MultiBench (Liang al., 2021a). We randomly selected 20 points from the test split of CMU-MOSEI dataset. The original CMU-MOSEI dataset designed for 7-way sentiment classification (-3 to +3), but follow the preprocessing in MultiBench and convert it into binary classification problem
36

---

Published as a conference paper at ICLR 2023
MulbViz Simulation Experiment
Smmeet
Pemmarto bel of Soid
inteeeeeeen anreng Lame fems vew
Reliven comph
Endee boye
Figure 25: Simulation experiment for CMU-MOSEL: Sample Features page. Best viewed zoomed in and in color.
37

---

Published as a conference paper at ICLR 2023
Table 5: Local faithfulness evaluations on visualizing cross-modal interactions. We study the correlation between model performance and what cross-modal interactions are picked up by the model. Across 2 models on CLEVR, we find that the ability to capture cross-modal alignment, as judged by MULTIVIZ, correlates strongly with final task performance.
Model
Top alignment Top 2 alignment
Method
Dataset
Model
Accuracy
accuracy
accuracy
Second Order Gradient
MDETR (Kamath et al., 2021)
99.5%
55.8%
80.7%
CLEVR CNN+LSTM+SA (Johnson al., 2017)
68.5%
21.2%
32.7%
(where -1, -2, are "Negative" and 0,1,2,3 are "Positive"). For each of the points, we run MULTIVIZ analysis and visualization: for U stage we show first order gradient analysis on image, audio and text (for image and audio, we compute gradient on each feature on each timestep, resulting in 2d heatmap, while for text we just have a 1d heatmap), and we also show processed video where we add bounding boxes around the visual features the model picked up (such as facial landmarks, facial expressions, lip movements, eye gaze, etc); for C stage we perform second order gradient analysis with selected words on image and audio; for R we show first order gradient on image, audio and text with respect each representation feature; for Ry, each representation feature we present 3 data points that maximally activates the feature, and also show first order gradient visualization for each; for P stage we show the "graph" on the right that ranks the top 5 representation features from Sparse Linear Model analysis as well as their respective weights. The webpage organization the same as the webpage for VQA with the Overview page (Figure 24) and Features pages (Figure 25).
Within each of the five groups, on each of the 20 points, human annotators are asked to predict what the model (MulT) predicts given a website containing some all of the stages of analysis visualizations (depending on the group's setting), Before each human annotator starts, they are taught how to interpret each analysis visualization, and the annotator needs to finish the 20 points on their own. We then compute average accuracy and inter-rater agreement score (Krippendorff's s alpha) within each group.
As shown in Table 2, in general, human annotators were able better predict the model's predictions when they were given more information, as the groups that got more information almost always end up with both higher average accuracy and higher inter-rater agreement. Moreover, human annotators were able to get perfect accuracy and agreement in settings (4) and (5), showing that including global analysis Rg provides enough information to simulate model predictions.
D.2 -ROSS-MODAL INTERACTIONS
In order to verify local faithfulness of interpreting cross-modal interactions, we take closer look at the qualitative and quantitative performance of our proposed second-order gradient method.
D.2.1 CLEVR
One gold standard for evaluating visualizations of cross-modal interactions involves using CLEVR (Johnson et al., 2017) (for image question answering). because in this dataset we are given ground truth bounding boxes of each object and there are often modal alignments that are obvious and without any controversy. We picked two representative models: MDETR (Kamath al., 2021), which is near-perfect (with 99.5% accuracy); and CNN+LSTM+SA (Johnson al., 2017), which was the best model amongst the baselines included in the paper that introduced CLEVR dataset (Johnson et al., 2017). We randomly selected 52 ground -tuuh alignment pairs, all of which aligns between a phrase in the question (1-4 words) and the one single object in the image. Then, for each pair, we compute the first-order gradient of each word with respect to the sum of all entries in the prediction logit vector, sum up the absolute gradients of the words in the phrase, before taking the gradient of each pixel with respect to the sum. We end up with a second-order gradient (SOG) on each pixel. Then, we then compute the average absolute SOG per pixel within bounding boxes of each object (given by CLEVR). We compute metrics: alignment picked up by top 1 bounding box (how often does the aligned object match with the bounding box with the highest average SOG) and alignment picked up by top bounding box (how often does the aligned object match with one the bounding boxes with top 2 highest average SOG).
38

---

Published as a conference paper at ICLR 2023
(model makes mistake)
other small shiny thing the the tiny yellow shiny object is what color?
What color object that is the some shope the large brown matte thing?
Figure 26: Examples of cross-modal interactions on CLEVR captured by proposed second-order gradient method. Left: an example with the MDETR model, where it picks up the correct cross-modal interaction and predicts the correct answer. Right: example with the CNN+LSTM+SA model, where does not pick up the correct cross-modal interaction and results in an incorrect answer. (Within each of the two example, the image the left side is heatmap on absolute second order gradient for each pixel, and on the right shows top 2 bounding boxes with highest average absolute second order gradient per pixel. top box in red, top 2 box in blue).
girl front pink food tray getting bike helmet woman.
dog with white facial and chest markings standing chest high water.
Three small dogs, two white and block and white, sidewalk.
dog with brown ears running sidewalk.
Examples of moosl interactions captured by ViLT Flickr-30k dataset discovered Figure 27:
by our proposed second-order gradient approach.
We show the results Table We found that under near-perfect setting (MDETR) where is safe assume that the model actually picks up all ground-truth alignments, our method was able to pick up over 80% of the alignments using top-2 bounding boxes, thus indicating that our method is quite faithful to the model's actual prediction process. Moreover, we found that CNN+LSTM+SA, which a relatively simple late fusion model with relatively poor performance, was much less likely to pick up the correct alignments according to our method, which makes sense. Below, we show examples of when the model picks up or is unable to pick up the ground- truth alignments in Figure 26.
D.2.2 FLICKR-30K
In addition, perform similar experiment for Flickr-30k image-text retrieval by modifying the above approach slightly. We select 20 image-text pairs from the annotated dataset, and for each of them take between 8-15 phrases find the second-order-graddiet (SOG) each pixel. We take the ground-truth boxes from Flickr30k Entities (Plummer et al., 2015) and calculate the average SOG for given object per pixel across the available boxes for the object. Additionally, we match the phrase against ground -rouu phrase annotations to find relevant boxes. Finally, we calculate what percentage of the objects were recovered by double gradient from the ground-truth annotations, any. For ViLT (Kim et al., 2021) model, we observe that second-order gradient is able to do with 44% matching accuracy, as compared to 34% using random matching (see some examples of detected interactions in Figure 27). For CLIP, the matching performance is worse (35%) as the gradients very scattered across examples, making it hard to localize one particular object (see some examples of detected interactions in Figure 28). Both these findings indicate potential future directions towards quantifying intermediate cross modal interactions learned by a model beyond looking at final task performance.
39

---

Published as a conference paper at ICLR 2023
white dag with brown ears running sidewalk.
handed individual with orange scarf and face covering uses small knife sculpt piace
Two boys, two girls, strapped in and ready for pork ride.
mon standing on street with suitcase front while onother man bends down to look at what displayed
Figure 28: Examples of cross-modal interactions captured by CLIP Flickr-30k dataset discovered by our proposed second-order gradient approach.
Qunstion: what is the tab made of) Id: 554 Correct ancwer glass Pred Arswor: glass
localeeplaration
0u3
200
400
Figure 29: Example R example with Unimodal LIME explanation given to annotators in the representation feature interpretation experiment.
D.3
REPRESENTATION INTERPRETATION
We now take deeper look to check that MULTIVIZ generates accurate explanations of multimodal representations. Using local and global representation visualizations, can humans consistently assign interpretable concepts in natural language previously uninterpretable features?
D.3.1 VQA 2.0
For VQA 2.0 dataset, we perform representation interpretation experiment, where we give human annotators some visualizations
particular representation feature and ask them to describe what concept they think that feature represents. We found 15 human annotators (with same qualifications those in model simulation experiment), and divide them into 3 groups of 5. Each group is given different setting (with different amounts of MULTIVIZ visualizations available):
RN R only, i.e. one random example and Unimodal LIME explanation on the example with respect to this example. See Figure 29 for example.
Re + R2 (no viz): In addition R with LIME, we also provide R, (top 3 examples that maximizes the feature's value and top 3 examples that minimizes the feature's 's value), but no LIME visualizations for Ry. See Figure 30 for example.
3. R, Rg: Same setting 2, but we also provide Unimodal LIME visualizations for all examples Rg. See Figure for example.
We gave the same 13 representation features all 15 human annotators, where the first feature serves as an example and the other 12 are the ones we actually record for the experiment. The instructor first explains each annotator what each visualization means, and then goes over the first feature
40

---

Published as a conference paper at ICLR 2023
Figure 30: Example of Rg examples without Unimodal LIME explanation given to annotators under Setting 2 together with R visualizations in the representation feature interpretation experiment. Note that the left examples are the ones that minimize the feature's value, while the right 3 examples are the ones that maximize the feature's s value.
Figure 31: Example of R, examples without Unimodal LIME explanation given to annotators under Setting 3 together with R, visualizations in the representation feature interpretation experiment. Note that the left 3 columns are the ones that minimize the feature's value, while the right 3 columns are the ones that maximize the feature's value. Within each column, from top bottom in order: the example data point, unimodal image LIME visualization, and unimodal text LIME visualization. Best viewed zoomed and in color.
together. Then, the annotator must write down a concept for the other 12 features on their own. We also ask each annotator rate confidence of 1-5 how confident they that this feature indeed represents this concept.
Once we have collected all 180 annotations (15 annotators each on 12 features), we manually cluster these into 29 distinct concepts that show in Figure 32. For example, annotations like "things to wear". "t-shirts" and "clothes" all belong to "clothes" concept; all color-related annotations belong to "colors" concept; "material question", "made-of question" and "material of object" all belongs to "material" concept. We then compute inter-rater agreement score on each feature within each group of 5 annotators using Krippendorff's s alpha with 29 possible categories. We report both inter-rater agreement and average confidence in Table 3.
As shown in Table 3, as we give annotators more information, they were able to assign concepts more consistently (higher inter-rater agreement) and more confidently (higher average confidence score). Under setting with full MULTI Viz visualizations on feature representations, the annotators completely agreed with each other 7 out of 12 features, which is really impressive since there
41

---

Published as a conference paper at ICLR 2023
none
what
abjects abovelunder relationship
plants
street
modifiers idee
numbers
vegetable
bathroom
baby
plane
sky
motorcycle
pizza
plate
lotchen
rooms in house
identifying things
near building objects
material
brush teeth
something covering
airport
people
on plato
food
poles
Figure 32: The 29 concepts that grouped all 180 annotations (12 features 15 annotators) into in order to compute categorical inter-rater agreement.
Feature annotated (confidence)
Minor
2. Brush feeth (3) Bethroom
R
Material question
None (1) Agreement 0/5, Avg confidence = 1.88
Question text asks what something made of (4)
2. Images are scones from inside home, question asks Household materials
R₁ Rg (no viz)
4. Queetion asking what certain objects are made out of (S)
5. Non-wood kitchen components Agreement Avg confidence
(6)
2. Manufacturing Materials (5)
Rj Rg
The question asks you to identily the material of something (5)
Materials (5)
Questions asking materialsscopposltnn (5)
Agreement S/5, Avg confidence 5.0
Figure 33: An example of visualizations given to for cases R₁, + Rg (no viz), and R; Rg for feature interpretation (Section 3.2 in the main paper), along with actual feature concepts annotated by the users.
so many possible concepts annotators could assign to each feature. Therefore, this shows that our visualizations, i.e. R₂ and Ry, really helps humans to better understand what concept (if any) that each feature in representation represents, and that Rg examples and visualizations are especially helpful.
concrete example: In Figure 33, we show a concrete example of human annotators using MULTI- VIZ to assign concepts to feature representations in multimodal models trained VQA 2.0. We show the information provided to users each of the 3 ablation cases as part of the experiment, along with the actual user annotations from the study:
In RN, we only provide the original seed datapoint and show visualizations of unimodal and cross-modal interactions with respect to a feature for that datapoint. Using just local information, annotators struggle to identify the concept captured by the feature z, with disagreement between 'mirror,, 'buushing teeth', 'bathroom', 'material', and 'none", each with relatively lower confidence. Indeed, any of the concepts are present in the image and question, which makes it hard choose precise one.
In R R, (no viz), we provide both original seed datapoint (local analysis), along with similar datapoints that also maximally activate the feature (global analysis), for 3 datapoints in total. Using both local and global information, users are better able to identify the commonalities between all datapoints which all active feature leading to 3/5 users identifying the concept as "asking about material'. However, the remaining 2 users answered 'household objects/componnetss, which is another valid concept shared across those datapoints.
42

---

Published as conference paper at ICLR 2023
Image concopt: metallic objects (CLEVR)
Multimodal concept: asking about material (CLEVR)
lorge
sphere sylnder?
object of? side the right side of the cylhder??
yellow sonere thing that
large yollow right is mode of what moterial?
Figure 34: Examples of auman-annotated concepts using MULTIVIZ on feature representations, We find that the features separately capture image-only, language-only and multimodal concepts.
Image concept: vehicles detected (MM-IMDb)
and forgiving communist leader decides send a young worker...
Sometime twenty-
..o generation charismatic drivers who roced on the
first century, the space craft
Zero-X is embarking on a manned mission Mars...
edge, risking their lives during
Formula 1's deadliest period...
Figure 35: More examples of human-ammaana annotated concepts using MULTIVIZ on feature representations. We find that the features separately capture image-only, language-only, and multimodal concepts.
In R + Rg. we show both local and global analysis (so datapoints in total), in addition to the visualizations of unimodal and cross-modal interactions with respect to a feature for all datapoints. With all pieces of information, all 5/5 users identified the concept as 'asking about material'. Providing visualizations helps to resolve ambiguity feature interpretation the text importance identifies words like 'counter', 'countertop', and 'wall', along with the image crossmodal interactions highlighting these entities, which leads to high agreement and confidence among annotators in identifying the 'material" concept.
D.3.2 CLEVR
few examples of interpreted representations are shown in Figure 34, in addition the examples in Figure of the main paper.
D.3.3
MM-IMDB
A few examples of interpreted representations shown in Figure 35, in addition the examples in Figure the main paper.
D.4 ERROR ANALYSIS
In this section, we conduct an experiment to see if human annotators will be able to categorize the reasons why the model fails predict the correct answer.
D.4.1 SETUP
We present three categories of errors:
Unimodal perception error: The model fails to recognize certain unimodal features aspects. (For example, in Figure 5 top left example, the FRCNN object detector was unable to recognize the thin red streak as an object).
2. Cross-modal interaction error: The model fails to capture important cross-modal interactions such as aligning words question with relevant parts or detected objects image. (For example, in Figure 5 first one in middle column, the model is erroneously aligning "creamy" with the piece of carrot).
3. Prediction errors: The model is able to perceive correct unimodal features and their cross-modal interactions, but fails to reason through them to produce the correct prediction. (For example, in Figure 5 top right example, the model was able to both perfectly identify the chair with object
43

---

Published as a conference paper at ICLR 2023
detector and associate it with the word "chair" in the question (as shown by second-order sedddegga analysis), but the model was still unable to reason with the given information correctly to predict the correct answer).
For each of the 2 datasets we used in this experiment (VQA and CLEVR), we found 10 human annotators and divide them into 2 groups of 5, one group for each setting: (1) under MULTIVIZ setting, for each data point, the human annotator is given access to full MULTIVIZ webpage as well as live Second-Order Gradient (i.e. the human annotator may request to compute second order gradient for a specific subset of words in the question, and he will be presented with the resulting second order gradient result); (2) under No Viz setting, the human annotator is given nothing but the original data point, the correct answer and the predicted answer. Each human annotator needs to classify each point into one of the three categories above, and they are also asked to rate their confidence in categorizing the error scale of 1-5.
D.4.2 VQA 2.0
In this experiment, we perform error analysis on VQA 2.0 with LXMERT. We first randomly selected 24 data points which the model got wrong, and then we ask 10 human annotators to categorize each point into one of the 3 categories above (5 annotators under MULTIVIZ setting and 5 annotators under No Viz setting). The webpage that the human annotators under MULTIVIZ setting sees is the same as the ones described in Appendix B. In addition, since the LXMERT prediction pipeline is differentiable with respect to the detected objects by FRCNN object detector but not with respect to each pixel in the original image, the human annotators under MULTIVIZ setting will also be given all the bounding boxes of objects detected by FRCNN and also which ones have the highest second order gradient with respect to the specific words they picked. See Figure 36 for an example of all bounding boxes detected by FRCNN as well as second-order gradient analysis results for LXMERT.
During the experiment, the instructor first informs the annotators what each of the 3 categories of errors mean, and then explains each part of the visualizations they are given (if under MULTIVIZ setting). Then the instructor goes over the first data point together with the human annotators, and the human annotators must categorize the remaining 23 points their own, and only those 23 points' annotations will count towards the final result.
The result for VQA error analysis experiment is shown in Table 3. As shown in the table, on average the human annotators are much more confident in categorizing each error, and also tend to agree with each other a lot more often when given MULTIVIZ compared No Viz. This shows that MULTIVIZ can indeed help humans identify types of errors within a multimodal model. In addition, human annotators from the MULTIV setting report that they can tell whether model is able perceive unimodal information correctly via U stage analysis as well as the bounding boxes produced by FRCNN, and they found second order gradient requested on specific words most helpful among all C stage visualizations (such as DIME) when determining if the model was able to find the correct cross-modal interactions. The data point presented in Figure 36 is one good example of this.
Error breakdown: Out of the 23 total errors, human annotators reported that on average 8.8 of them are category 1 (unimodal perception error). 6.8 of them are category 2 (cross modal interaction error), and 7.4 of them present in LXMERT is still caused by misunderstanding the basic unimodal concepts and mrosll alignments rather than high-level reasoning of the perceived information, and that one possible future direction for improving the model pipeline is to use better unimodal encoders (than FRCNN) and find out some way to force the model learn to align visual and text concepts correctly.
category 3 (prediction error). This suggests that the majority of errors
A concrete example: In Figure 37, we show a concrete example of human annotators using MULTI- Viz to perform error analysis on incorrect predictions made by trained models, specifically into one of 3 stages: failures in (1) unimodal perception, (2) capturing modal interaction, and (3) prediction with perceived unimodal and cross-modal information. We show the information provided to users in each of the 2 ablation cases, along with the actual user annotations from the user study:
1. No Viz does not provide the user with any information. Note that there are no intermediate stages we can ablate, since errors can occur at all stages, removing any stage from MultiViz by definition cripples its ability to detect errors at that stage. However, users still use their intuition to make most educated guess on which stage the model is likely to make error in. For example, if some odd object seems hard to detect, users tend to guess unimodal error, and if the prediction involves complex reasoning that is hard even for humans, users tend to guess prediction error.
44

---

Published as a conference paper at ICLR 2023
Ouesiieee there lits chesse thi place! 1383 Correct snewen no Arewer yes
100
200
335
150
100 200
800 700
200
Figure 36: Examples of second-order gradient request by human annotators during error analysis. Left: input image. Middle: all bounding boxes detected by FRCNN image encoder. Right: top bounding boxes with the highest second order gradient with respect to the word "pizza" (red is top 1, blue is top 2, green is top 3). this data point, we can clearly see that the model was able detect the pizza (as it was included in a bounding box by FRCNN) and was able to associate the pizza in the image with the word "pizza" the question (as shown by second order gradient analysis). Therefore, through MULTIVIZ visualizations, all 5 human annotators agreed that this point is category prediction error. Best viewed zoomed in and in color.
Error category annotated (confidence)
viz:
MultVViz:
50
1. Cross-modal (1)
1. Prediction
75
2. Unimodal (1)
2. Prediction (5)
Q: Is there lot of cheese on this pizza?
3. Prediction
3. Prediction
4. Unimodal (2)
4. Prediction (5)
125
Predicted answer: Yes Correct answer: No
5. Prediction
Prediction (3)
Agreement = 2/5
Agreement = 5/5
Avg confidence
Avg confidence
200
25-
viz:
MultiViz:
1. Unimodal (2)
1. Cross-modal (5)
Q: is this guy's jeans ripped?
2. Cross-modal (1)
Cross-modal (5)
300
3. Prediction (4)
3. Cross-modal (4)
Predicted
4. Cross-modal
(4)
Correct answer: No
5. Cross-modal (2)
5. Cross-modal (5)
150
Agreement 3/5
Agreement
175
Avg confidence = 2.2
Avg confidence 4,6
200
50
150
Figure 37: An example of visualizations given to users for error analysis on incorrect predictions made by trained models, specifically into one of stages: failures in (1) unimodal perception, (2) capturing cross-modal interaction, and (3) prediction with perceived unimodal and cross-modal information (Section 3.3 in the main paper), along with actual error categories annotated by the users.
MUlTIVIZ provides the user with the unimodal importance and mross-modal interactions visual- ized for that incorrectly predicted datapoint. In the top example, users can tell that the unimodal importance on 'cheese' and "pizza" are correct, along with the right image-text interaction high- lighting the bounding pizza around pizza. Hence, it is a prediction error, which all users agree on. In the bottom example, users can see that while 'man' and 'jeans' are unimodally highlighted correctly, none of the image-text interactions highlight the bounding box around the man's 's jeans, so they agree on a modal interaction error.
D.4.3 CLEVR
In this experiment, we perform error analysis CLEVR with CNN+LSTM+SA model. We first randomly selected 11 data points which the model got wrong, and then we ask 10 human annotators to categorize each point into one of the 3 categories above (5 annotators under MULTIV ViZ setting and 5 annotators under No Viz setting). The webpage that the human annotators under MULTIV VIZ setting sees is the same the ones described in Appendix B. In addition, the human annotators under
45

---

Published as a conference paper at ICLR 2023
Unimodal Cross-modal interactions
"Models pick cross-modal interactions fail Identifying color!"
Multimodal
Add torgeted exomples involving color.
representations
Multimodal
prediction
Figure 38: Model debugging: ask humans use MULTIVIZ visualizations and identify bugs that multimodal model exhibits. Following this, we will attempt to fix the bug given a fixed budget of additional datapoints that the model is allowed access to. If MULTIVIZ indeed helps humans to identify the correct reason for model failure, the targeted data given to the model should improve performance more so than a same amount randomly sampled data.
MULTIVIZ setting can request the second-order gradient analysis result on specific words phrases they pick, both the pixel-wise heatmap and top bounding boxes with the highest average absolute gradient per pixel (same procedure described in Appendix D.2). See the bottom half of Figure 26 for an example of second-order gradient analysis result CNN+LSTM+SA.
During the experiment, the instructor first informs the annotators what each of the 3 categories of errors mean, and then explains each part of the visualizations they are given (if under MULTIVIZ setting). Then the instructor goes over the first data point together with the human annotators, and the human annotators must categorize the remaining 10 points on their own, and only those 10 points' annotations will count towards the final result.
The result for CLEVR error analysis experiment is shown in Table 3. As shown in the table, on average the human annotators are much more confident categorizing each error, and also tend to agree with each other lot more often when given MULTIV V Iz compared to No Viz. This shows that MULTIVIZ can indeed help humans identify types of errors within a multimodal model.
Out of the 10 total errors, human annotators on average reported 6 of them belonging category (cross modal interaction error). This suggests that the major weakness of CNN+LSTM+SA is that it is not great at aligning phrases in text with the object the phrase refers to. This is expected because CNN+LSTM+SA late fusion model, which known to be not great at capturing low-level cross-modal interactions.
Error breakdown:
D.5 MODEL DEBUGGING
D.5.1 VQA 2.0
Following error analysis, we take a deeper investigation into one of the errors on a pretrained LXMERT (Tan and Bansal, 2019) model fine-tuned on VQA 2.0 (Goyal et al., 2017).
We compute the penultimate features (the input to the last linear layer in the classification head) of the set, and train a linear model that best maps the absolute values of these penultimate features to binary label where 0 means the original LXMERT model got this point right and 1 means the original LXMERT model got this point wrong. Then, we pick the top dimensions in the penultimate feature with the highest positive weight the linear model, and task human annotators inspect these neurons carefully through MULTIV1Z local and global representation analysis. Human consistently related questions asking about color,
annotators found that 2 of the 5 neurons which highlighted the model's failure to identify color correctly (especially blue). The model has an accuracy of only 5.5% amongst all blue-related points (i.c., either have blue as correct answer or predicted answer). and these failures account for 8.8% of all model errors. We show examples of such datapoints and their MULTIVIZ visualizations in Figure 6. Observe that the model is often able to capture unimodal and cross-modal interactions perfectly, but fails to identify color at the prediction stage.
In this section, we describe our initial attempt at fixing this color colaeerelated bug by adding targeted data in active learning scenario. If MULTIVIZ indeed provides accurate insights for model debugging, we should be able to improve model performance using less data as compared to a control experiment that adds randomly sampled data (see Figure 38).
46

---

Published as a conference paper at ICLR 2023
Table 6: Model debugging: we task 3 human users to use MULTIVIZ visualizations and highlight the bugs that a model exhibits (see Figure 38), and find 2 penultimate- layer neurons which highlighted the model's failure to identify color. The model has an accuracy of only 5.5% amongst all blue-related points (i.e., either have blue as correct answer or predicted answer), and these failures account for 8.8% all model errors. By providing the model with 500 additional datapoints asking specifically about color (in an active learning setup), we improve the model, especially on the targeted points.
Research area
QA
Dataset
VQA 2.0 (Goyal et al., 2017)
Model
LXMERT (Tan and Bansal, 2019)
Metric
largeted accuracy A Overall accuracy
Random
+1.4 0.3
0.1
Uncertainty (Lewis and Catlett, 1994)
+0.0 0.0
+0.1
MULTIVIZ color
+2.5 + 1.3
+..1
MULTIVIZ color
+27.5 1.9
+1.0
MULTIVIZ 2 color
30.5 + 4.9
+1.2 0.2
We first split the validation set (about 220K points) into 3 parts: the first 110K were called the V set (stands for "val"), the next 50K were called the U set (stands for "unfabeled"). and the last 60K were called the T test (stands for "test"). We simulating situation where in addition to the 450K training set, we have a labeled 110K validation set (V set), another 50K unlabeled points (U set), and 60K held-out test set (T set). Our goal is to debug or improve the given model (LXMERT) by selecting N points from U set to label and finetune the model with these N points.
We compare the following settings:
Random: We randomly sample N points from the set.
2. Uncertainty: A common active learning baseline which selects the top N datapoints from the U set that the model is uncertain about based on the entropy of its predicted label distribution (Lewis and Catlett, 1994; Lewis and Gale, 1994; Settles, 2009).
N 3. MULTIVIZ 2 color: For each of these 2 erroneous features, we picked 2 points from the U set that has the highest absolute values on the feature, and together these points form the N points related to color that we select from the U set. Note that we do not use label information about these additional datapoints.
4. MULTIVIZ. no color: Same as above, but we use 2 features that do not represent color.
5. MULTIVIZ 1 color: Same as above, but we use I feature that represents color and that does not represent color.
Under each of these active learning settings, we finetune the last layer of LXMERT with the N selected points from U set for one epoch (batch size 32, learning rate tuned to the best performance), and the result is evaluated out that LXMERT is particularly bad on data points that either have ground truth correct answer "blue" or the original LXMERT predicts as "blue", we define a subset of T set we call "bluelist" that contains all 1729 points in the T set that either have ground truth correct answer "blue" or the original LXMERT predicts as "blue". The original LXMERT only has a 6% accuracy on bluelist. We try each setting 10 times (with different random seeds) and report average and standard deviation on improvement accuracies on both the entire T set and bluelist over the original LXMERT.
the T set. In addition, since through MULTIVIZ analysis we found
We show these results in Table 6 and find that MULTIVIZ significantly improves upon either random or uncertainty-basel sampling measured by performance the overall VQA 2.0 test set. To obtain a deeper look at performance, we further evaluate performance on a targeted test set only containing questions asking about color (reflecting the main bug we found in the model). On this targeted test set, MULTIVIZ significantly improves performance by 30% as compared to only 1.4% for random sampling. Using more features related to color also improved performance: 27% with feature and 30% with both features. Surprisingly, we find uncertainty sampling had effect (0.0% since the model predicted these incorrect answers on color-related questions with high certainty, so none of these color-related questions were additionally introduced to the model.
D.6 SUMMARY OF TAKEAWAY MESSAGES
From these results, emphasize the main take-away messages:
47

---

Published as a conference paper at ICLR 2023
1. From the model simulation experiment, found that on all settings of datasets and models, human annotators were able to get higher accuracy and better agreement when given strictly more stages of visualization from MULTIVIZ. This suggests that each stage in MULTIVIZ is complementary to helping humans better understand the models' decision-making process. 2. Through deeper inspection of cross-modal interaction visualization, we showed that second-order gradient is faithful to what the model internally aligns most of the time (over 80% using top 2 alignment accuracy on MDETR).
3. From the representation interpretation experiment, we found that having both local and global representation visualizations helps human annotators assign interpretable concepts in natural language to deep features with higher confidence and agreement.
From the error analysis experiment, we showed that MULTIVIZ can help users locate the stage of model that caused the error when the model makes mistake, which provides insights for model error analysis and debugging.
5. Finally, showcase real-world model debugging case study: using each stage of MULTIVIZ to localize the error, we were able to locate a real bug in the HuggingFace Transformers LXMERT library.
E LIMITATIONS AND FUTURE DIRECTIONS
E. LIMITATIONS
We aware of some directions in which MULTIVIZ can still be improved and outline these for future work:
1. Large number of prediction classes: for complex tasks like VQA 2.0 where there are over three thousand prediction classes, lot of rarely used answer choices will get "sparsed out" in the Sparse Linear Model analysis (since setting their weights to 0 barely affects overall accuracy), which makes it difficult find related datapoints for local and global representation analysis. For example, Sparse Linear Model Analysis on LXMERT have zero weight from all representation neurons the rare answer choice "abstract", the five most important feature neurons are completely randomly selected.
Too few prediction classes: for VQA 2.0 subsets with 'yes/no' answer choices, we found that the final-layer activated features contain too much overlap to reliably visualize, and we have to extend MultiViz to rely on more intermediate-layer features. We added this experiment in Appendix B.3. Overall, MULTIVIZ (like general MI. models), work best with reasonable number of prediction classes, such as those in multimodal emotion recognition, standard multiple-choice multimodal question answering, and others.
Model requirements: Currently the two requirement of models is that they have categorical outputs (classification) and we can easily compute gradients via AutoGrad. The classification requirement is so that we can visualize given specific model outputs (e.g.. word answers, emotion categories, video categories). For regression, we extend MultiViz via discretizing the output space into categorical outputs. The second requirement enables us to perform first and higher-order gradient analysis, which means that we cannot currently support some neuro-symbolic multimodal architectures that have discrete steps (c.g., parsing and executing the question as program (Mao al., 2019)) in the middle of the model that prevents gradient flow. We plan to extend Multi Viz via approximate gradients such as perturbation or policy gradients to handle these cases.
Visualization testing: We spent lot of time into finding and training users. We carefully found users (who are not the authors and are not part of the same research groups) that have or are working towards a graduate degree in a STEM field and have knowledge of ML models. We showed them training video describing how Multi can be used before each study session (see Appendix D for all experiment and user study details). Consequently, our user studies span over 60 hours of human testing on close to 100 total datapoints, which has enabled us draw preliminary conclusions regarding the efficacy of multiple proposed stages towards model understanding and debugging. Future work can explore more standardized ways human-in-the-loon interpretation and debugging of multimodal models, and we hope that MULTIVIZ can provide the initial data, models, tools, and evaluation as a step in this direction.
We plan to ensure the continual availability, maintenance, and expansion of MULTIVIZ Several immediate directions include new interpretation algorithms and holistic evaluation of interpretation methods.
48

---

Published as a conference paper at ICLR 2023
E.2 NEW CLASSES OF INTERPRETATION TOOLS
MULTIVIZ is designed to be modular and support interpretation tools at each stage. While we have explored some directions, we plan include the following methods in future work:
E.2.1 CROSS-MODAL INTERACTIONS
There have been several attempts at building multimodal models that are interpretable by design, with particular focus on cross cross-modal interactions. Many of these involve parameterizing cross-modal interactions through attention models (Hu and Singh, 2021; Lu et al., 2019) or graph- based mod- els (Liang et al., 201 8b; Zadeh et al., 2018). As result, there have been several approaches to study these specific types of cross-modal interactions, such as M2Lens (Wang et al., 2021), an interactive visual analytics system to visualize multimodal models for sentiment analysis through both unimodal and cross-modal contributions, and VL-InterpreT (Aflalo et al., 2022), an interactive visualization tool for interpreting vision-language transformers. We plan to include these in MULTIVIZ compare black-bo box post-hoc interpretation versus models interpretable by design.
E.2.2 MULTIMODAL PREDICTION
Beyond linear prediction, we also plan to investigate integrating neural networks with decision trees (Wan et al., 2020) generalize linear reasoning into one based compositionality defined by decision tree, or other hierarchical prediction processes (Andreas et al., 2016; Yi et al., 2018)).
E.3 EVALUATING INTERPRETABILITY
Progress towards interpretability is challenging to evaluate (Chan et al., 2022; Dasgupta et al., 2022;
Jacovi and Goldberg, 2020; Shah et al., 2021; Srinivas and Fleuret, 2020). Model interpretability (1) highly subjective across different population subgroups (Arora et al., 2021; Krishna et al., 2022), (2)
requires ligh-dimensionll model outputs as opposed to nww-dimensional prediction objectives (Park et al., 2018), and (3) has desiderata that change across research fields, populations, and time (Murdoch al., 2019). We plan continuously expand MULTIV through community inputs for metrics to evaluate interpretability methods. Some metrics we have mind include those for measuring faithfulness, as proposed in recent work (Chan al., 2022: Dasgupta et al., 2022; Jacovi and Goldberg. 2020; Madsen et al., 2021; Shah et al., 2021; Srinivas and Fleuret, 2020).
E.4 ENGAGEMENT WITH WORLD STAKEHOLDERS
Finally, we have plans for engagement with real- world stakeholders to evaluate the usefulness of these multimodal interpretation tools. We plan engage these stakeholders in the healthcare domain to evaluate interpretability on the MIMIC dataset and those in the affective computing domain evaluate interpretability on the CMU-MOSEI dataset. We also refer the reader recent work examining the issues surrounding real-world deployment of interpretable machine learning (Bhatt al., 2020; Chen et al., 2022; Krishna al., 2022).
49


Reviews:
[{"review_id": "rI9qeNTpPG", "reviewer": "Reviewer_Xfxj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce multi-modal Visualization tool MultiViz. \nIt provides indicator to the input image and text pairs visualizing: unimodal importance, cross-modal interactions, multi-modal representations, and multi-modal prediction.\nFor unimodal importance, MultiViz uses uni-modal feature attribution method.\nFor cross-modal interactions, it builds upon statistical non-additive interaction.\nFor multi-modal representations, it perform local and global representation analysis.\nHere, local analysis visualizes unimodal and cross-modal interactions (area and/or words) that activate a feature, while\nglobal analysis informs user of similar datapoint that also maximally activate that feature (activate similar concept).\nFor multi-modal prediction, it approximates the linear prediction model with linear combination of penultimate layer features.\n\nThe authors tested the model on a number of real-world multimodal tasks: fusion, retrieval, question answering; datasets VQA 2.0, MM-IMDb CMU-MOSEI.\nThey set up model simulation to help participants in the experiment predict the outcome of the model.\nThe experiments shows that with MultiViz visualized evidence, people do see better predicting the output of the model.\nThey show that the global and local analysis individually is inferior to having both analyses.\nThe users are more confident and higher agreement among participants \nThe authors also perform qualitative interview to hear that MultiViz is valuable in performing the task.\n", "summary_of_the_review": "The technique will be very useful for debugging tools in multi-modal neural network research.", "strength_and_weaknesses": "Strengths: The method is quite complete in its implementation of the multiple levels of feature visualization. The review of other methods is also valuable.\n\nWeaknesses: \nIt does not talk about what is still needs to be done. A few examples of the most difficult cases it is able to properly visualize, and what is not. They have a section of error analysis. However, this is the error of the model, not the error of the visualizer.\nI would like to have a paragraph or two on how hard it is to set up a visualization testing, whether the models should have certain characteristics, etc.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lzXybAyQ4q", "reviewer": "Reviewer_1CKW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a framework/tool for understanding and evaluating multimodal networks. This includes four main parts: unimodal, cross modal, multimodal (representations) and multimodal (predictions). They also present a novel interpretability methods for a subset of these. Finally, they evaluate this framework with a user study. ", "summary_of_the_review": "Overall this is a solid and interesting paper. I would like to see more screenshots and descriptions of the actual tool, and more related work re: end-to-end interpretability systems.", "strength_and_weaknesses": "Strengths:\n- Ambitious and multidisciplinary (standard interpretability, HCI, visualization) approach to ML interpretability. It's impressive that they both developed a new method and fully tested it in an end-to-end scenario, incorporated with prior methods.\n- Well thought-out experiment setup-- as you discuss, interpretability methods are hard to evaluate, and they authors took a creative and interesting approach via the human agreement setup.\n- Good coverage of different modalities. Often when people say \"multimodal\", they just mean text and images, so it's cool to see that you also used video, audio, time series, and tabular data. \n\nWeaknesses:\n- While I do really appreciate how multidisciplinary this paper is, the downside is that it straddles a few different areas without fully committing to any of them. I'm curious about a deeper evaluation of the new interpretability method on its own (e.g., looking at the sanity checks for saliency maps paper [https://arxiv.org/abs/1810.03292[ and seeing what aspects could be relevant). On the other hand, if this is a systems paper, why are the screenshots of the actual tool hidden in the appendix? I would expect a figure 1 screenshot of the actual system that is being used.\n- Similarly, if this is a paper about an end to end interpretability UI tool, I'd expect more related work in this area (e.g., from Jeff Heer's lab https://homes.cs.washington.edu/~jheer/, the Language Interpretability Tool https://arxiv.org/abs/2008.05122, etc)\n- It wasn't immediately clear what were the author's contributions vs prior work, in terms of specific interpretability methods. It's fine to have this be a system that incorporates past methods and makes them actually usable, but it would have been better if this was clearer (e.g., a table of methods and their origins would have been helpful.)\n- Algorithm 1 seems a bit overly technical-- it seems like it would have been clearer just have this as bullet points (or just a screenshot of the UI annotated with what features each interpretability method was using) \n- 64 pages is way too long. If something is after page 20 of an appendix, it probably shouldn't be in the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JcpdL3vrYz", "reviewer": "Reviewer_eaCz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes MULTIVIZ which is a method to analyze the workings of multi-modal models. The authors propose to break down the visualization / analysis problem into four primary pillars, namely: \nunimodal importance: how each modality contributes to the downstream tasks\ncross-modal interactions: how the multiple modalities interact with each other \nmulti-modal representations: how uni-modal and cross-modal interactions are represented in features and \nmulti-modal predictions: how the features are transformed to make a decision. \n\nThe idea seems to be interesting, and the visualizations do show some merits in analyzing models by visualizing the cross-modal interactions. The paper claims to be generic in terms of modalities, models, tasks and research areas. ", "summary_of_the_review": "Overall, an interesting paper but needs more experimentation validation on newer models. The development is too verbose as I mentioned before and needs more mathematical rigor to bring out the connection between the proposed analysis goals and what is exactly achieved by the formulation. ", "strength_and_weaknesses": "Analyzing multi-modal models is the need of the hour with the proliferation of these models in the research community. The proposed division of the problem into uni-modal and multi-modal pillars makes sense, but the paper seems to be using the penultimate layer of the models as the feature representation on which the analysis is based on. Why this layer and not some of the higher layers which are closer to the lower-level representation of the inputs is chosen is not described well in the paper. The figures 1 and 3 need more refinement for clear understanding of the intuitions being explained through them. \n\nThe mathematical development is extremely sparse and unsatisfactory. Definition 2 seems to be the backbone of the subsequent analysis, but no intuition has been developed as to why should this work. The sentence \"Specifically, given a model f, we first take a gradient of f with respect to an input word (e.g., x1 = birds), before taking a second-order gradient with respect to all input image pixels x2, which should result\nin only the birds in the image being highlighted (see Figure 2 for examples on real datasets).\" should be explained with more rigor. The authors seem to believe that the connection from the math to the claim should follow as is. \n\nThe prediction part assumes a linear model which is rarely the case for the SOTA models. The inner representations are also outputs from non-linear activations functions. All these mathematical complexities are not even mentioned in the visualization effort. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IJy4KM3n9mQ", "reviewer": "Reviewer_L6AN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a framework they call as \"MultiViz\" aimed at visualizing the internal working of multimodal models. In my opinion, the major contribution from the paper is breaking down multimodal interpretability into 4 components: (1) unimodal contributions, (2) cross-modal interactions, (3) multimodal-representations, and (4) multimodal predictions. Through comprehensive experiments, the authors demonstrate that the 4 components are complimentary to each other. In addition, they show that MultiViz can be effectively enable users to (1) simulate model predictions, (2) identify interpretable concepts with features, (3) perform error analysis, and (4) debug models. ", "summary_of_the_review": "MultiViz is an interesting approach with wide applications. I believe the paper, along with the promised open source software that will be maintained will be quite impactful. ", "strength_and_weaknesses": "The paper is extremely well written, and very clear! Overall, Multiviz is a pretty innovative approach, and I think it will have great impact in the field. \n\n1. The authors have designed and performed thorough experiments to validate the usefulness of the 4 components they proposed. I also appreciate the authors acknowledging in the last section the tother breakdowns could be possible. \n2. The authors have demonstrated the usefulness in a wide variety of tasks, and on different models. \n3. The supplementary materials looks thorough (full disclosure: haven't read even half of it!), and looks very detailed. \n\nWhile not weakness, I do have some comments for the authors:\n1. I really like the error analysis experiments that the authors performed. I wonder if MultiViz can be used to understand why certain models perform better than the other? For example, in a given task, can we interpret why transformer models (or any other class), outperform CNNs? Is there any systematic shortcoming of models that can be uncovered using this approach? \n2. I urge the authors to incorporate data and models from other domains as well. For example, medical AI maybe a good area where MultiViz tool could actually make an strong impact and could be put to use immediately. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster