Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a framework they call as "MultiViz" aimed at visualizing the internal working of multimodal models. In my opinion, the major contribution from the paper is breaking down multimodal interpretability into 4 components: (1) unimodal contributions, (2) cross-modal interactions, (3) multimodal-representations, and (4) multimodal predictions. Through comprehensive experiments, the authors demonstrate that the 4 components are complimentary to each other. In addition, they show that MultiViz can be effectively enable users to (1) simulate model predictions, (2) identify interpretable concepts with features, (3) perform error analysis, and (4) debug models. 
strength_and_weaknesses: The paper is extremely well written, and very clear! Overall, Multiviz is a pretty innovative approach, and I think it will have great impact in the field. 

1. The authors have designed and performed thorough experiments to validate the usefulness of the 4 components they proposed. I also appreciate the authors acknowledging in the last section the tother breakdowns could be possible. 
2. The authors have demonstrated the usefulness in a wide variety of tasks, and on different models. 
3. The supplementary materials looks thorough (full disclosure: haven't read even half of it!), and looks very detailed. 

While not weakness, I do have some comments for the authors:
1. I really like the error analysis experiments that the authors performed. I wonder if MultiViz can be used to understand why certain models perform better than the other? For example, in a given task, can we interpret why transformer models (or any other class), outperform CNNs? Is there any systematic shortcoming of models that can be uncovered using this approach? 
2. I urge the authors to incorporate data and models from other domains as well. For example, medical AI maybe a good area where MultiViz tool could actually make an strong impact and could be put to use immediately. 


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The experiments and work is novel and of high quality. The authors have provided enough information for reproducibility. 
summary_of_the_review: MultiViz is an interesting approach with wide applications. I believe the paper, along with the promised open source software that will be maintained will be quite impactful.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a framework/tool for understanding and evaluating multimodal networks. This includes four main parts: unimodal, cross modal, multimodal (representations) and multimodal (predictions). They also present a novel interpretability methods for a subset of these. Finally, they evaluate this framework with a user study. 
strength_and_weaknesses: Strengths:
- Ambitious and multidisciplinary (standard interpretability, HCI, visualization) approach to ML interpretability. It's impressive that they both developed a new method and fully tested it in an end-to-end scenario, incorporated with prior methods.
- Well thought-out experiment setup-- as you discuss, interpretability methods are hard to evaluate, and they authors took a creative and interesting approach via the human agreement setup.
- Good coverage of different modalities. Often when people say "multimodal", they just mean text and images, so it's cool to see that you also used video, audio, time series, and tabular data. 

Weaknesses:
- While I do really appreciate how multidisciplinary this paper is, the downside is that it straddles a few different areas without fully committing to any of them. I'm curious about a deeper evaluation of the new interpretability method on its own (e.g., looking at the sanity checks for saliency maps paper [https://arxiv.org/abs/1810.03292[ and seeing what aspects could be relevant). On the other hand, if this is a systems paper, why are the screenshots of the actual tool hidden in the appendix? I would expect a figure 1 screenshot of the actual system that is being used.
- Similarly, if this is a paper about an end to end interpretability UI tool, I'd expect more related work in this area (e.g., from Jeff Heer's lab https://homes.cs.washington.edu/~jheer/, the Language Interpretability Tool https://arxiv.org/abs/2008.05122, etc)
- It wasn't immediately clear what were the author's contributions vs prior work, in terms of specific interpretability methods. It's fine to have this be a system that incorporates past methods and makes them actually usable, but it would have been better if this was clearer (e.g., a table of methods and their origins would have been helpful.)
- Algorithm 1 seems a bit overly technical-- it seems like it would have been clearer just have this as bullet points (or just a screenshot of the UI annotated with what features each interpretability method was using) 
- 64 pages is way too long. If something is after page 20 of an appendix, it probably shouldn't be in the paper.

clarity,_quality,_novelty_and_reproducibility: This paper is mostly clearly written (with the exception that I would have liked more explicit information on what interpretability methods came from where). There are no obvious logical holes or anything-- my complaints are mostly around novelty, and fleshing out the "systems" side a little more.
summary_of_the_review: Overall this is a solid and interesting paper. I would like to see more screenshots and descriptions of the actual tool, and more related work re: end-to-end interpretability systems.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce multi-modal Visualization tool MultiViz. 
It provides indicator to the input image and text pairs visualizing: unimodal importance, cross-modal interactions, multi-modal representations, and multi-modal prediction.
For unimodal importance, MultiViz uses uni-modal feature attribution method.
For cross-modal interactions, it builds upon statistical non-additive interaction.
For multi-modal representations, it perform local and global representation analysis.
Here, local analysis visualizes unimodal and cross-modal interactions (area and/or words) that activate a feature, while
global analysis informs user of similar datapoint that also maximally activate that feature (activate similar concept).
For multi-modal prediction, it approximates the linear prediction model with linear combination of penultimate layer features.

The authors tested the model on a number of real-world multimodal tasks: fusion, retrieval, question answering; datasets VQA 2.0, MM-IMDb CMU-MOSEI.
They set up model simulation to help participants in the experiment predict the outcome of the model.
The experiments shows that with MultiViz visualized evidence, people do see better predicting the output of the model.
They show that the global and local analysis individually is inferior to having both analyses.
The users are more confident and higher agreement among participants 
The authors also perform qualitative interview to hear that MultiViz is valuable in performing the task.

strength_and_weaknesses: Strengths: The method is quite complete in its implementation of the multiple levels of feature visualization. The review of other methods is also valuable.

Weaknesses: 
It does not talk about what is still needs to be done. A few examples of the most difficult cases it is able to properly visualize, and what is not. They have a section of error analysis. However, this is the error of the model, not the error of the visualizer.
I would like to have a paragraph or two on how hard it is to set up a visualization testing, whether the models should have certain characteristics, etc.  
clarity,_quality,_novelty_and_reproducibility: Paper is clear and concise in the main article, with further information available in the appendices.
summary_of_the_review: The technique will be very useful for debugging tools in multi-modal neural network research.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes MULTIVIZ which is a method to analyze the workings of multi-modal models. The authors propose to break down the visualization / analysis problem into four primary pillars, namely: 
unimodal importance: how each modality contributes to the downstream tasks
cross-modal interactions: how the multiple modalities interact with each other 
multi-modal representations: how uni-modal and cross-modal interactions are represented in features and 
multi-modal predictions: how the features are transformed to make a decision. 

The idea seems to be interesting, and the visualizations do show some merits in analyzing models by visualizing the cross-modal interactions. The paper claims to be generic in terms of modalities, models, tasks and research areas. 
strength_and_weaknesses: Analyzing multi-modal models is the need of the hour with the proliferation of these models in the research community. The proposed division of the problem into uni-modal and multi-modal pillars makes sense, but the paper seems to be using the penultimate layer of the models as the feature representation on which the analysis is based on. Why this layer and not some of the higher layers which are closer to the lower-level representation of the inputs is chosen is not described well in the paper. The figures 1 and 3 need more refinement for clear understanding of the intuitions being explained through them. 

The mathematical development is extremely sparse and unsatisfactory. Definition 2 seems to be the backbone of the subsequent analysis, but no intuition has been developed as to why should this work. The sentence "Specifically, given a model f, we first take a gradient of f with respect to an input word (e.g., x1 = birds), before taking a second-order gradient with respect to all input image pixels x2, which should result
in only the birds in the image being highlighted (see Figure 2 for examples on real datasets)." should be explained with more rigor. The authors seem to believe that the connection from the math to the claim should follow as is. 

The prediction part assumes a linear model which is rarely the case for the SOTA models. The inner representations are also outputs from non-linear activations functions. All these mathematical complexities are not even mentioned in the visualization effort. 
clarity,_quality,_novelty_and_reproducibility: The paper seems to repeat its primary claims quite a few times. The abstract, Sec 1 and Sec 2, each repeat the four analysis pillars proposed in this paper almost verbatim. The paper is hard to understand because of the sparsity of mathematical development and also on the continuous usage of the actual tool images which might not be the most optimized interface to show the results and insights succinctly. 

The paper seems to be a novel effort in capturing multi-modal visualizations but needs more refinement both in the tool (images shown) and the overall development as well. 

Reproducibility: I am not sure whether the results can be reproduced easily. One major blockage I see is the use of LXMERT paper as the base for the experiments. This is an older paper with a good code base available. More analysis on newer models would at least rest my reproducibility doubts.
summary_of_the_review: Overall, an interesting paper but needs more experimentation validation on newer models. The development is too verbose as I mentioned before and needs more mathematical rigor to bring out the connection between the proposed analysis goals and what is exactly achieved by the formulation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a MultViz framework that can for visualizing multimodal models from four perspectives: unimodal importance, cross-modal interaction, multimodal representations, and multimodal prediction. Through experiments are done to validate the usefulness of the four components on a wide range of multimodal tasks and models. The paper is also well written.  
The rebuttal and revision greatly improved the paper quality and addressed all the concerns from the reviewers (one reviewer raised their score from 3 to 6). The AC agrees with the reviewers that this work is a valuable contribution to the multimodal ML community and would like to see it presented in the ICLR conference. 
justification_for_why_not_higher_score: Though a tool is provided for visualizing and understanding multimodal models, the reviewers found that it lacks insightful discussion about how to build more interpretable multimodal models.
justification_for_why_not_lower_score: All the reviewers unanimously appreciated the contributions of this paper, which sheds lights on understanding and interpreting multimodal models from multiple perspectives.
Decision: Accept: poster