# Leveraging Concept Annotations for Trustworthy Multimodal Video Interpretation through Modality Specialization

Elisa Ancarani, Julie Tores, Rémy Sun, Lucile Sassatelli, Hui-Yin Wu,
Frederic Precioso

<figure><img src="image_1.png" /></figure>

## To cite this version:

Elisa Ancarani, Julie Tores, Rémy Sun, Lucile Sassatelli, Hui-Yin Wu, et al.. Leveraging Concept Annotations for Trustworthy Multimodal Video Interpretation through Modality Specialization. MM '25:The 33rd ACM International Conference on Multimedia, Oct 2025, Dublin Ireland, France. pp.11-19, 10.1145/3746277.3760415 . hal-05393708

HAL Id: hal-05393708

https://hal.science/hal-05393708v1

Submitted on 9 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

Latest updates: https://dl.acm.org/doi/10.1145/3746277.3760415

RESEARCH-ARTICLE

# Leveraging Concept Annotations for Trustworthy Multimodal Video Interpretation through Modality Specialization

ELISA ANCARANI, University of Côte d'Azur, Nice, Provence-Alpes-Côte d'Azur, France

JULIE TORES, University of Côte d'Azur, Nice, Provence-Alpes-Côte d'Azur, France

RÉMY SUN, University of Côte d'Azur, Nice, Provence-Alpes-Côte d'Azur, France

LUCILE SASSATELLI, University of Côte d'Azur, Nice, Provence-Alpes-Côte d'Azur, France

HUIYIN WU, University of Côte d'Azur, Nice, Provence-Alpes-Côte d'Azur, France

FRÉDÉRIC PRÉCIOSO, University of Côte d'Azur, Nice, Provence-Alpes-Côte d'Azur, France

Open Access Support provided by:

University of Côte d'Azur

<figure><img src="image_8.png" /></figure>

PDF Download

3746277.3760415.pdf

09 January 2026

Total Citations: 0

Total Downloads: 95

Published: 27 October 2025

Citation in BibTeX format

MM '25:The 33rd ACM International
Conference on Multimedia

October 31, 2025

Dublin, Ireland

Conference Sponsors:

SIGMM
# Leveraging Concept Annotations for Trustworthy Multimodal Video Interpretation through Modality Specialization

**Elisa Ancarani**

Université Côte d'Azur, CNRS, I3S
Sophia Antipolis, France
elisa.ancarani@univ-cotedazur.fr

**Julie Tores**

Université Côte d'Azur, CNRS, Inria, I3S
Sophia Antipolis, France
julie.tores@univ-cotedazur.fr

**Rémy Sun**

Université Côte d'Azur, Inria, CNRS, I3S
Sophia Antipolis, France
remy.sun@inria.fr

**Lucile Sassatelli**

Université Côte d'Azur, CNRS, I3S
Sophia Antipolis, France
lucile.sassatelli@univ-cotedazur.fr

**Hui-Yin Wu**

Université Côte d'Azur, Inria
Sophia Antipolis, France
hui-yin.wu@inria.fr

**Frédéric Precioso**

Université Côte d'Azur, CNRS, Inria, I3S
Sophia Antipolis, France
frederic.precioso@inria.fr

<figure><img src="image_9.png" /><figcaption>Figure 1: Combining input modalities can sometimes lead to wrong decisions. In this movie clip, a visual-only model correctly identifies the objectifying dimension, whereas a model combining text and visual modalities fails to do so. Concept annotations reveal that objectification occurs only in the visual modality, the addition of text indiscriminately caused the error. This paper shows that proper combinations of models specialized for different modalities yield the best trade-off between task performance and trustworthiness as depicted in the right hand plot.</figcaption></figure>

## Abstract

Multimodal datasets usually come as multimodal data annotated for a certain construct (such as depression). However, for such tasks of video interpretation, models must not only make accurate predictions, but make them for the right reasons. Ensuring model trustworthiness is however hampered by the lack of per-modality information. We consider the case of a recently introduced dataset for the video interpretation task of detecting objectification, annotated for this end task along with multimodal explanatory concepts that provide per-modality labels. With such additional knowledge, we study how to design models with both high task accuracy and modality trustworthiness. We first introduce the MSpecF framework articulating and fusing a spectrum of variably specialized models, and two trustworthiness metrics. We show that modality-specialized models

generally maximize trustworthiness, and maximize task accuracy for confident modalities. For less certain modalities, task accuracy is maximized by non-specialized models. We show that the full fusion of specialized models MSpecF(All*) achieves advantageous trade-offs between task accuracy and trustworthiness compared to other fusion choices. This work shows that rich per-modality annotations of moderate-size datasets allow to make more trustworthy models, essential for applications such as supporting social scientists in analyzing complex social constructs.

## CCS Concepts

* **Computing methodologies** → **Neural networks**; *Scene understanding*; *Social and professional topics* → User characteristics.

## Keywords

Video interpretation, Multimodality, Trustworthiness, Explanatory concept annotation

## ACM Reference Format:

Elisa Ancarani, Julie Tores, Rémy Sun, Lucile Sassatelli, Hui-Yin Wu, and Frédéric Precioso. 2025. Leveraging Concept Annotations for Trustworthy Multimodal Video Interpretation through Modality Specialization. In *Proceedings of the 1st International Workshop on Cognition-oriented Multimodal Affective and Empathetic Computing (CogMAEC '25)*, October 27–28, 2025, Dublin,

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

CogMAEC '25, Dublin, Ireland

© 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM.

ACM ISBN 979-8-4007-2059-8/2025/10

https://doi.org/10.1145/3746277.3760415
*Ireland*. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3746277.3760415

# 1 Introduction

Detecting high-level semantic constructs in videos, such as anomaly [31], depression [22], social message [6] or character portrayal [8] poses significant challenges for multimodal learning: models must not only make accurate predictions, but make accurate predictions for the right reasons. That is, they must be trustworthy, and as transparent as possible. Existing approaches to multimodal trustworthiness estimate per-modality confidence using only the end task label [17, 38]. However, the lack of ground truth about which modality (or combination thereof) drives the final prediction limits our ability to assess model reasoning and hampers the development of trustworthy and fair models. These requirements have made eXplainable AI (XAI) an important research area for tasks that involve complex semantic constructs. In XAI, rationales are referred to as concepts, which are interpretable units of knowledge that motivate the final prediction.

To assess or enforce models to make decisions based on interpretable concepts, researchers have created concept-rich datasets and explored concept supervision, particularly in vision. These datasets pair task labels (e.g., bird species in [5], malignancy in [10]) with explanatory concepts (e.g., yellow beak in [5], presence of a nodule in [10]). However, densely annotated datasets with such concepts remain rare, though they would enable a reliable evaluation of misattributions and biases, an important step toward improving model trustworthiness.

This is specifically the case for video-interpretation tasks, where tasks are complex and content tedious to annotate, while trustworthiness and bias mitigation are key for models automating the detection of social constructs to assist social scientists to make valid estimations on large corpora and draw valid conclusions [13, 26]. There is pressing need for such fine-grained modality based analysis. Fig. 1 shows precisely that combining input modalities can introduce new issues rather than resolve them. While multimodal integration is often expected to enhance performance, it can instead lead to confusion. For example, in Fig. 1, a specialized visual-only model correctly identifies objectification in the clip, whereas a more generic video-text model is confused by the added textual input.

In this article, we study the trustworthiness of multimodal models in the light of per-modality ground-truth labels. We consider a recent dataset, ObyGaze, introduced by Tores et al. [8] for the task of detecting when a character in a movie is objectified. Its multimodal extension, MObyGaze [33], consists of movie clips annotated both with objectification labels but also with modality-specific explanations that indicate whether the perception of objectification arises from visual, auditory or textual cues. We therefore contribute answers to the following question: *with a dataset annotated with multimodal explanatory concepts and an interpretative task label, how to design models with high performance on both dimension: task accuracy and modality trustworthiness?*

Our contributions are as follows:

* We first introduce a framework to study how to flexibly specialize models in the light of modality-specific ground-truth labels. We name our framework Modality-Specialized Fusion, MSpecF. This

framework encompasses a spectrum of modality-specialized to modality-generic models. The former are models with a single input modality and trained with the modality-specific labels, while the latter are regular multimodal models trained on the end task label. We then introduce two trustworthiness metrics, *modality accuracy* and *modality misattributions*, building on the knowledge of modality-specific labels. While specific, these metrics follow the same principles as recent works on trustworthiness quantification when modality-specific labels are not available [37].

* We investigate how different model combinations balance task accuracy and modality trustworthiness on the recently released the MObyGaze dataset. For the most certain modality (vision), fully specialized unimodal models perform best in terms of task accuracy and trustworthiness (modality accuracy). For less certain modalities (text then audio), modality-specialized models offer higher trustworthiness, while modality-specific label (one input modality, but supervision with the end task label) yields better task accuracy.

* When assessing trustworthiness across modality pairs, a similar pattern emerges: fusing fully specialized unimodal models yields the best task accuracy and trustworthiness for the most certain pair (vision and text), whereas for pairs involving the least certain modality (audio), task accuracy is highest with less specialized models that take both modalities as input.

* Finally for all three modalities, we show that fusing unimodal, bimodal and trimodal models each trained on their respective modality-specific labels, which we call MSpecF(All*), achieves best results in task accuracy and trustworthiness (modality accuracy) in most cases, which suggests notable differences and complementarity in the mechanisms learned. Considering the trustworthiness metric of modality misattributions, we find that MSpecF(All*) achieves advantageous trade-offs between task accuracy and trustworthiness compared to other fusion choices.

While large-scale datasets have driven significant advancements across many areas of AI, there is a growing recognition, echoed by Paullada et al. [28], that smaller, carefully curated datasets are crucial for developing socially responsible and trustworthy AI. This work underscores the value of producing not necessarily large-scale, but thoughtfully designed datasets, developed in close collaboration with domain experts. Such an approach not only addresses key societal challenges, like exposing patterns of disparity or injustice by quantifying the presence of a complex construct, but also fosters the development of models that can be trusted to yield confident and meaningful conclusions at inference time.

The article is organized as follows. Section 2 reviews related work on multimodal fusion, trustworthiness, and concept-based explainability. Section 3 introduces the multimodal objectification dataset and task. Section 4 describes our MSpecF framework and trustworthiness metrics. Section 5 reports experimental results, and Section 7 concludes. Code is available at<fnref n="1" />

# 2 Related Works

We position our work with respect to the areas it combines: multimodal fusion, trustworthiness and concept-based explainability.

<footnote n="1">¹https://src.koda.cnrs.fr/2rx2gvqs/mspecf-framework</footnote>
## 2.1 Multimodal fusion

Fusing information from diverse modalities is usually envisioned in gradual ways from the joint processing of input in a common model, often denoted as early fusion, up to the late fusion of the decisions of modality-specific models, through total or partial fusion of per-modality features, which we expose for videos. Recent early fusion approaches concatenate visual and textual (caption) tokens from movie clips and input them into a shared transformer network to be learned for, e.g., movie genre classification [32] or audio-description generation [15]. When considering benefiting from pre-trained LLMs, which are known to be difficult to fine-tune efficiently, prompt-tuning can be used where adapter layers are learned per modality before concatenating the inputs in a prompt fed to a frozen LLM [16]. Early fusion has been shown to enable robustness to noise in input modalities. Another way to intermediate combine features of modalities into a common transformer network is through cross-attention, where modalities do not have symmetric roles, as in [14]. Simple intermediate fusion with concatenation and fully-connected layers has also been considered for movie-related tasks [20, 34]. It is also generic to gate at later stages the features from individual modalities or some partial combination, often combined with cross-attention [18, 21]. Early, intermediate and late fusion can also be combined. For example, to detect depression, the approach in [22] integrates visual, audio, text and physiological signals by combining them both with early and late fusion through a voting mechanism based on interpretability scores from [24]. The MM-AU approach in [6] is closer to ours in that it considers detecting social message from advertisements, and considers hybrid fusion of pair-specific models, with a hard-coded late fusion of the individual decisions. In MM-AU, individual models early fusing pairs of modalities are pre-trained on the label of the video (even though not all modalities are fed to these models to best approach the label). This approach of pre-training modality-specific models has been recently considered in [2, 29, 39]. [2] entirely relies on partially hard-coded fusion for multimodal video retrieval and shows that modality-specific models allow to reach reference performance. Image-text classification is targeted in [29] by enabling alignment-based fusion by pre-training individual modality-specific models on the end task, to then enforce stronger alignment between both modalities based on the accuracy of each model. While in those works modality-specific models are pre-trained with the end task label resulting from all modalities, in this work we pre-train models with modality-specific labels, and name them modality-specialized models, composing the MSpecF framework.

## 2.2 Trustworthiness

Multimodal trustworthiness is often considered as coping with noise and uncertainty [17, 25, 38]. [38] considers learning feature confidence to inform fusion, while [17] proposes modality-specific classifiers enabling to assess feature-level and modality-level informativeness for each sample, to then fuse the modality representations based on these levels.

Recently, benchmarks have been created to assess the trustworthiness of multimodal LLMs [35, 37]. Specifically, [37] presents a benchmark with four dimensions including truthfulness, which is defined as inherent deficiency and misguided mistakes: the former

corresponds to basic model performance while the latter are errors due to an interfering modality. These correspond to the trustworthiness metrics we introduce in Sec. 4.3: modality accuracy and modality misattributions.

## 2.3 Explainability and concept-based datasets

Interpretability in machine learning refers to the extent to which a human can understand the reasons behind a model's decision [27]. Without such explanations, it becomes challenging to examine and validate predictions, especially in complex tasks where the goal is to identify high-level patterns. As noted by [3] focusing only on optimizing for the end-task, without considering how decisions are made, can lead to unintended gaps in the model's reasoning. This may introduce uncertainty and reduce trust, particularly in such sensitive applications where understanding *why* a conclusion emerges is just as important as the conclusion itself. However, only scarce works have considered multimodal interpretability. A prominent one is [24] proposing a method to analyze the behavior of multimodal models in terms of importance granted to each modality for each sample, how different modalities interact and how they are composed in the prediction. For the single visual modality however, concept-driven explainability is an active area of research, where models are trained based on detailed annotation of image data: not only is the image annotated with the construct of interest (e.g., malignancy of skin lesion [10] or bird species [5]), but the annotator also provides which concepts are present, amongst a pre-existing list, that explain their label decision. In this work, we consider a newly released dataset on multimodal character objectification in movies (see Sec. 3.1), annotated with concepts spanning the visual, textual and audio modalities, hence enabling us to obtain per-modality labels, unlike existing approaches. Explainability in concept-based models is generally assessed along with so-called task accuracy: a model is evaluated on the target classification task, along with the interpretability of its decision, measured by various metrics [4, 36]. In this article, we evaluate models with a similar methodology, with task accuracy and modality trustworthiness, the definition of which is inspired by [37] as discussed above.

The novelty of the present work is its positioning at the crossroads of multimodal fusion, trustworthiness and concept-based explainability. We seek how to best leverage a new concept-annotated multimodal dataset, the first of the sort TTBK, to design multimodal models and assess them on task accuracy and modality trustworthiness.

# 3 Dataset and Task

In this section, we first introduce the specific task and dataset we consider, and then define the modality-specific datasets on which we base our contributions to assess and improve the per-modality trustworthiness of multimodal models. These basic elements are shared with the setting introduced in the study [1].

## 3.1 Data and task for objectification detection

Our goal is to investigate how per-modality concepts can support high-level video interpretation tasks, focusing on objectification detection, a recently introduced computer vision task [8]. In movies, objectification arises when a character is portrayed as a passive
object of desire or service rather than an active narrative agent [8]. This phenomenon, often referred to as "male gaze", draws from media studies, gender studies, and psychology, and plays a central role in shaping societal perceptions of gender. This high-level construct is conveyed through multimodal cues: visual (e.g., type of clothing, submissive posture, body-focused, perspective-taking framing), textual (e.g., suggestive dialogue), and auditory (e.g., giggling voice). Understanding and quantifying objectification and how it is unevenly applied across demographic groups, is of strong interest in the social sciences. Tores et al. [8] propose to automate objectification detection in movies by introducing the ObyGaze (objectifying gaze) dataset, recently expanded in a multimodal version, MObyGaze, available online<fnref n="2" />, with all the details in [8, 33]. MObyGaze includes movie clips annotated with both a task label (presence or absence of objectification) and modality-specific explanations indicating which modalities (visual, textual, auditory) contribute to the perception of objectification. These rich annotations support rigorous trust assessment in complex multimodal settings, where only a subset of modalities may be relevant. Task labels mark either negative samples (no objectifying cues) or positive ones (presence of objectifying factors). When objectification is present, the dataset distinguishes between partial and full perception, but in our work, we treat both as positive and rely on the modality-specific annotations to identify contributing channels. An example is shown in Fig. 1. MObyGaze contains 5,783 video clips from 20 movies densely annotated by human experts, with 58% labeled positive and 42% negative. Although more frequent, positive clips are shorter, making up only 37% of the total duration compared to 63% for negative clips.

## 3.2 Concept Modality Specific Dataset (CSMD)

The availability of the explanatory concepts over all three modalities enables our objective of assessing how trustworthy multimodal models can be and how to best benefit from concept annotation to combine task accuracy and per-modality trustworthiness. For such assessment and model design, we need to adapt the data for the specific individual or combined modalities fed as model input. We call these datasets Concept Modality Specific Datasets (CSMDs). Therefore, we consider a specific dataset for each case: for each modality M (video, text or audio), M-CMSD is made of positive samples that are the clips where the M modality is annotated as contributing to the presence of objectification, the negative samples being the rest. Similarly, for each set of modalities M₁ and M₂, the positive clips of the M₁M₂-CSMD are those where M₁ or M₂ (or both) contribute to the presence of objectification. In contrast, the original MObyGaze dataset is denoted Concept Modality Agnostic Dataset (CMAD). The constitution of these datasets are represented in Fig. 2. Unimodal and multimodal models can be either trained on CMAD, or on their respective CMSD. Assessing them for task accuracy is done by inferring on CMAD, while assessing their trustworthiness with respect to a given (set of) modalities is made by inferring on the test set of the corresponding CMSD.

<figure><img src="image_10.png" /><figcaption>**Figure 2:** The Concept Modality Agnostic Dataset (CMAD) contains all positive samples (partial and full perception of objectifying factors) across modalities, whereas the Concept Modality Specific Dataset (CMSD) filters these to include only those with modality-specific concepts.</figcaption></figure>

## 4 Models and methodology

We present our methodology to identify to what extent should we specialize models by exploiting modality-specific labels (from generic to modality-specialized, as outlined in Fig. 3a), and how to flexibly combine different levels of specialization to best perform in task accuracy and modality trustworthiness.

We start by training a pool of models with different degrees of modality specificity and even specialization as discussed in Sec. 4.1 and Fig. 3a. We then introduce our MSpecF fusion scheme to combine these trained models into a final decision (see Sec. 4.2 and Fig. 3b). Afterwards, we introduce task accuracy and two trustworthiness metrics in Sec. 4.3.

### 4.1 Training a pool of models with different degrees of modality specialization and specificity

Our study requires comparing and combining models with different approaches to the multimodal inputs. To this end, we train 13 different models that range from fully modality generic to very specialized for a single modality as illustrated in Fig. 3a. Our exploration centers around two main axis of modality specialization: the specificity of model input modalities, and the specialization of the models for the modality of choice during training.

*Modality specific models.* We consider 7 model types across visual, text, and audio modalities, ranging from fully modality-specific (unimodal) to fully modality-generic (multimodal) models.

We first train hyper-specific models that only deal with one modality. These models provide a clear picture of how each modality can contribute to the final model decision on its own. While the performance of these unimodal models is expected to be lower than that of more complex multimodal models, they are also free from any interference from other modalities. In this sense, these specific unimodal models are the most modality-trustworthy in our study, as their decisions rely solely on their single input modality.

On the other extreme, we train a modality generic model that takes as input the three modalities indiscriminately. This model's performance is expected to be the best if it were able to make the

<footnote n="2">²https://anonymous.4open.science/r/MObyGaze-F600/</footnote>
<figure><img src="image_11.png" /><figcaption>**Figure 3:** Overview of our analysis's two main components. We work with a pool of models that use the input modalities differently, and aggregate them to benchmark the obtained results. From left to right, the models increase in complexity—from specialized (single-modality models trained on CMSD) to the most generic (all three modalities trained on CMAD) with the spectrum of intermediaries relying on the varying degrees of specialization from the more specialized models.</figcaption></figure>

most of the complete modality information it is given. It will however suffer the most from interference between input modalities, and might therefore be the least trustworthy of the models trained in this study. As a middle ground, we also consider bimodal models that take as input two input modalities: they mix information from two input modalities but are unaffected by the one excluded input modality. The performance and trustworthiness are therefore expected to fall between specific and generic models. This is an important point of reference to compare to unimodal models and fully multimodal models.

**Specializing models on CMSD datasets.** We take advantage of MObyGaze's rich annotations to further refine the specialization of our models for specific modalities. Indeed, we train two versions of each model *f*: a *modality-specialized model* *f*\* trained on the relevant CMSD (modality-level labels) and a *modality-specific model* *f* trained on CMAD. Note that the CMSD and CMAD datasets are the same when all input modalities are considered: AVT and AVT\* are the same model. As noted in Sec. 3.2, the training dataset has significant influence on how well models fit their input modalities: training a video only model on a Concept Modality Agnostic Model (CMAD) would cause indirect interference from other modalities in the information learned by the model. Separating these specialized and specific models should therefore bring further insights into the complementarities and interference between modalities.

**General model architecture.** All of our models follow the same global architecture. The inputs are tokenized (following a fixed modality specific protocol) and fed to a Perceiver IO [19] architecture to extract features. These features are then processed by a multi-layer perceptron with 1 hidden layer to output a prediction logits for objectification detection. The model is optimized to minimize a binary crossentropy loss between these output logits and the ground truth objectification labels. The Perceiver IO architecture is a standard transformer-based feature extractor for multimodal data: (multimodal) input tokens are used to enrich learned latent

queries through a cross-attention scheme, these latent queries are then refined through self attention layers before being used in turn to enrich decoder queries through a final cross attention scheme that pools information extracted from the latent queries.

## 4.2 MSpecF: aggregating Modality Specialized model decision through selective Fusion

This paper's central proposition is that models specifically tailored to different (groups of) modalities provide different insights on complex problems (objectification detection in this study). We study here how the different models trained in Sec. 4.1 can be leveraged and combined with our proposed MSpecF framework.

We propose MSpecF, a framework designed to aggregate Modality Specialized model decision through selective Fusion. MspecF($f_1, ..., f_n$) takes as input the (concatenated) individual predictions of *n* selected models $f_1, ..., f_n$ and feeds them to a multi-layer perceptron (MLP) with one hidden layer to obtain a final aggregated prediction. The MLP is trained on the respective CMSD of input if it fuses *f*\* models, otherwise on CMAD (e.g. text and video on Fig. 3b).

This approach accounts for dependencies between individual predictions by combining the different model predictions into a non-linear layer before synthesizing them into a final prediction. In principle, ensembling [23] should lead to correcting some mistakes of individual models and yield to a more reliable prediction. However, the summation of predictions remains a linear operation and does not in contextualize each individual prediction with regards to those of other models. MSpecF's non-linear structure allows it to identify the relevant predictions, correlations between predictions and draw a final conclusion that goes beyond a simple consensus. This should ideally allow the model to strike a balance between the low trustworthiness of generic models and the lower performance of less task-accurate but more trustworthy specialized models.

**Picking models to aggregate.** While MSpecF provides a method to aggregate the predictions of differently specialized (or specific)
models, we also need to choose which models should be aggregated. To provide the most representative view of the question, we consider 3 main strategies: picking only one model, picking specialized (or specific) unimodal models of the relevant input modalities, and aggregating all models together. We start by looking at the individual performance of a model f. In this case, MSpecF(f) is actually equivalent to directly using the prediction of the model f: we will directly refer to the picked model f and use its predictions in our experiments. This solution is useful to assess the quality of the individual models and provide points of comparison between them.

To compare modality-specialized and generic models, we focus on the MSpecF(T*, A*, V*) aggregation that combines text-, audio-, and video-only models when all modalities are available. This approach aims to maintain high trustworthiness while enhancing task performance by taking advantage of the complementary strengths provided by each modality.

## 4.3 Metrics of trustworthiness and task accuracy

Having introduced the models and our MSpecF fusion framework, we now present the evaluation metrics. Our goal is to leverage modality-specific labels to assess multimodal models along two dimensions: task accuracy and modality trustworthiness.

**Task accuracy:** the performance of a model on the task label, that is to predict the objectification label of the clip, as described in Sec. 3.1. This corresponds to run inference on CMAD. For this binary classification problem, we report AUC-PR for positive samples, AUC-PR for negative samples (swapping labels) to emphasize the importance of specificity for this task, and macro AUC-PR computed as the sum of both with a weight of 0.5.

**Trustworthiness:** we define the trustworthiness of a model as its capacity to make the right decision for the right reason(s). We leverage the knowledge of the explanatory concepts, providing per-modality labels for each clip. We denote the presence of concept modalities in a clip as $(v=0/1, t=0/1, a=0/1)$. We define two metrics of model trustworthiness, inspiring on [37], as described in Sec. 2.

The first metric is **modality-accuracy**, the accuracy for predicting per-modality labels. It corresponds to run inference on the CMSD of the input modality(ies) of the model, as explained in Sec. 3.2. Specifically, we evaluate classification performance against ground-truth modality labels using AUC-PR scores, we report results for positive classes, negative classes, and macro-averaged across all classes to avoid threshold-dependent biases. The second metric is complementary and analyses inference on subsets of the model's input modalities, to quantify how much the model deviates owing to non-relevant modalities. For this, we set the relevant metric to be either video or text. Assuming it is video, we can express when a model combining video, text and audio, e.g., AVT, deviates owing to irrelevant modalities, by counting the number of decision shifts of the model with respect to the reference model V:

* in the cases where model AVT is wrong but model V is right for the right reason (hence AVT is wrong due to text or audio). Denoting true and false positives and negatives as TP, TN, FP and

FN, these cases are counted as:

$$ shifts_{Vright} = |TN \ of \ V \cap FP \ of \ AVT|_{set=(v=0,t=0,a=0)} + |TP \ of \ V \cap FN \ of \ AVT|_{set=(v=1)} $$

* in the cases where AVT is right but for the wrong reasons (V is wrong while video is present and AVT predicts positive, hence owing to the other modalities as V is wrong):

$$ shifts_{Vwrong} = |FN \ of \ V \cap TP \ of \ AVT|_{set=(v=1,t=0,a=0)} $$

We similarly express shifts$_{T}$right and shifts$_{T}$wrong in reference to the text modality with the T model.

We finally define a level of **modality misattributions** of a multimodal model as:

$$ MisAttr = \frac{shifts_{Vright} + shifts_{Vwrong} + shifts_{Tright} + shifts_{Twrong}}{|(v = 0, t = 0, a = 0) \cup (v = 1) \cup (t = 1)|} \quad (1) $$

# 5 Experimental evaluation

In this section, we experiment with the spectrum of modality-specific and modality-specialized to modality-generic models combined within our MSpecF framework. We assess the performance over the dimensions of task accuracy and modality trustworthiness. This section is structured to answer the following questions:

* **RQ1:** How do modality-specialized models, benefiting from modality-level supervision, perform in terms of task accuracy and modality trustworthiness?

* **RQ2:** How to best ensemble modality-specialized and modality-specific models to achieve more interesting trade-off between task accuracy and trustworthiness?

**Setting:** all input features are extracted using frozen pre-trained encoders. Visual data is processed with X-CLIP [9], extracting features from 16-frame windows with a stride of 4. Audio segments are divided into 2-second chunks and encoded via AST [11] pre-trained on AudioSet [12]. Subtitles are tokenized and encoded using a BERT-based model. To adapt to fixed input sizes, audio and visual sequences are normalized to a maximum sequence length: zero-padded if short, or uniformly downsampled by segment-wise averaging if long. All modality-specific models, whether unimodal or multimodal (with concatenated inputs), use the same PerceiverIO [19] backbone, producing a decoder latent of shape (batch size, 8, 128). Each model has ~1.1M parameters (950K in the encoder, 150K in the decoder). For fusion, logits are concatenated, then passed through a compact MLP (< 130 params). To accommodate moderate dataset size, we train using 5-fold cross-validation over the 20 MObyGaze movies (4 test, 2 validation per fold). Class imbalance is addressed via random oversampling. Models are optimized using AdamW ($lr=10^{-3}$, weight decay=$10^{-5}$) with 10% warm-up, binary cross-entropy with logits loss, ReduceLROnPlateau scheduler (patience=5), and early stopping (patience=15).

All results in the following subsections will highlight the best in bold and the second best underlined.

## 5.1 Unimodal modality-specialized and modality-specific models

We begin our experimental analysis by quantifying the performance of the unimodal modality-specialized and modality-specific models
both in terms of task performance and trustworthiness. This will help contextualize later multimodal results and directly address our first research question (RQ1).

Table 1 represents the task accuracy and trustworthiness (modality accuracy metric) of the modality-specialized (T*, V*, A*) and modality-specific (T, V, A) models. E2E is the same architecture as MSpecF(T,V,A) but which has been pre-trained end-to-end from scratch, and whose resulting unimodal models are then frozen and adapted with the same MLP with CMSD supervision.

First, we observe that the gap between random and the modality-specialized models reveals a clear hierarchy in the modalities; Visual is the most certain, followed by Text then by Audio that is the least certain. V* achieves the highest task accuracy (0.784) and trustworthiness (0.757). For Text, modality-specialized model T* leads to substantial gains in trustworthiness (+36.5% Pos AUC-PR, +6.2% Neg AUC-PR) over modality-specific model T, even though the latter slightly edges out in task accuracy.

For Audio, models perform near chance. Unimodal branches from E2E multimodal models underperform hyper-specialized and hyper-specific counterparts, generalizing poorly alone despite full training data. This suggests modality interference and highlights the value of specialized supervision. These findings underscore the structural asymmetry of the MObyGaze dataset with heterogeneous certainty over the modalities (prevalence of visual content, text more sparsely informative, audio least present). This motivates the next section, where we investigate how to strategically combine models to harness complementary modality strengths and enhance both task accuracy and trustworthiness.

**Answer to RQ1:** unimodal models fully specialized (supervision with the modality-level label) achieve best performance both in terms of task accuracy and trustworthiness (modality accuracy) for the most certain modality, which is vision. For the less certain, text then audio, modality-specialized models achieve best trustworthiness but best task accuracy is obtained with modality-specific label (supervision with the end task label).

<table><caption>**Table 1:** Unimodal modality-specialized (CMSD) and modality-specific (CMAD) models, with 95%-confidence intervals in parenthesis.</caption><thead><tr><th rowspan="2">Modality</th><th rowspan="2">Strategy</th><th colspan="2">Trustworthiness (modality acc.)</th><th colspan="2">Task acc. (CMAD Inference)</th></tr><tr><th>Pos. AUC-PR</th><th>Neg. AUC-PR</th><th>Pos. AUC-PR</th><th>Neg. AUC-PR</th></tr></thead><tbody><tr><td rowspan="4">Text</td><td>T* (CMSD)</td><td><strong>0.366</strong> (0.323, 0.410)</td><td><strong>0.880</strong> (0.856, 0.904)</td><td>0.664 (0.631, 0.697)</td><td>0.445 (0.416, 0.474)</td></tr><tr><td>T (CMAD)</td><td><strong>0.268</strong> (0.232, 0.304)</td><td>0.828 (0.797, 0.860)</td><td><strong>0.701</strong> (0.673, 0.730)</td><td><strong>0.601</strong> (0.571, 0.631)</td></tr><tr><td>E2E</td><td>0.216 (0.181, 0.250)</td><td>0.791 (0.760, 0.823)</td><td>0.587 (0.545, 0.628)</td><td>0.421 (0.385, 0.457)</td></tr><tr><td>Random</td><td>0.226 (0.194, 0.258)</td><td>0.785 (0.754, 0.815)</td><td>0.609 (0.578, 0.640)</td><td>0.405 (0.374, 0.436)</td></tr><tr><td rowspan="4">Visual</td><td>V* (CMSD)</td><td><strong>0.676</strong> (0.654, 0.699)</td><td><strong>0.757</strong> (0.736, 0.778)</td><td><strong>0.784</strong> (0.769, 0.799)</td><td><strong>0.710</strong> (0.680, 0.740)</td></tr><tr><td>V (CMAD)</td><td><strong>0.646</strong> (0.626, 0.665)</td><td>0.743 (0.724, 0.763)</td><td>0.773 (0.754, 0.792)</td><td>0.706 (0.683, 0.728)</td></tr><tr><td>E2E</td><td>0.573 (0.549, 0.598)</td><td>0.639 (0.607, 0.671)</td><td>0.696 (0.666, 0.726)</td><td>0.572 (0.532, 0.612)</td></tr><tr><td>Random</td><td>0.485 (0.462, 0.508)</td><td>0.532 (0.510, 0.554)</td><td>0.609 (0.578, 0.640)</td><td>0.405 (0.374, 0.436)</td></tr><tr><td rowspan="4">Audio</td><td>A* (CMSD)</td><td>0.191 (0.163, 0.220)</td><td><strong>0.825</strong> (0.802, 0.848)</td><td>0.600 (0.565, 0.636)</td><td><strong>0.412</strong> (0.382, 0.442)</td></tr><tr><td>A (CMAD)</td><td>0.192 (0.171, 0.213)</td><td><strong>0.827</strong> (0.799, 0.854)</td><td><strong>0.607</strong> (0.580, 0.633)</td><td><strong>0.421</strong> (0.385, 0.458)</td></tr><tr><td>E2E</td><td><strong>0.221</strong> (0.184, 0.258)</td><td>0.776 (0.745, 0.807)</td><td>0.597 (0.559, 0.635)</td><td>0.403 (0.374, 0.431)</td></tr><tr><td>Random</td><td>0.185 (0.162, 0.209)</td><td>0.821 (0.799, 0.843)</td><td>0.609 (0.578, 0.640)</td><td>0.405 (0.374, 0.436)</td></tr></tbody></table>

## 5.2 Multimodal specialization spectrum and fusion with MSpecF

We expand our previous analysis by systematically quantifying the performance of modality-generic versus modality-specialized and modality-specific under the MSpecF framework.

<table><caption>**Table 2:** Trustworthiness (modality accuracy) and task accuracy of paired modalities for different levels of specialization and fusion strategies, with 95%-confidence intervals in parenthesis.</caption><thead><tr><th rowspan="2">Modality</th><th rowspan="2">Strategy</th><th colspan="2">Trustworthiness (modality acc.)</th><th colspan="2">Task acc. (CMAD Inference)</th></tr><tr><th>Pos. AUC-PR</th><th>Neg. AUC-PR</th><th>Pos. AUC-PR</th><th>Neg. AUC-PR</th></tr></thead><tbody><tr><td rowspan="4">Text</td><td>TV* (CMSD)</td><td>0.746 (0.729, 0.763)</td><td>0.700 (0.675, 0.726)</td><td>0.768 (0.751, 0.785)</td><td>0.697 (0.670, 0.723)</td></tr><tr><td>MSpecF(T*, V*) (CMSD)</td><td>0.752 (0.733, 0.772)</td><td>0.707 (0.678, 0.737)</td><td>0.772 (0.754, 0.79)</td><td>0.706 (0.676, 0.735)</td></tr><tr><td>MSpecF(T*,V*) (CMSD)</td><td><strong>0.765</strong> (0.687, 0.737)</td><td><strong>0.72</strong> (0.711, 0.761)</td><td><strong>0.786</strong> (0.771, 0.802)</td><td><strong>0.718</strong> (0.688, 0.749)</td></tr><tr><td>Text</td><td>TV (CMAD)</td><td>0.737 (0.649, 0.758)</td><td>0.692 (0.666, 0.717)</td><td>0.759 (0.739, 0.779)</td><td>0.687 (0.662, 0.713)</td></tr><tr><td rowspan="4">Audio</td><td>MSpecF(TV,CMAD)</td><td>0.754 (0.732, 0.776)</td><td>0.710 (0.685, 0.735)</td><td>0.776 (0.754, 0.797)</td><td>0.708 (0.684, 0.732)</td></tr><tr><td>MSpecF(TV,TV) (CMAD)</td><td><strong>0.757</strong> (0.736, 0.779)</td><td><strong>0.713</strong> (0.686, 0.741)</td><td><strong>0.779</strong> (0.758, 0.8)</td><td><strong>0.711</strong> (0.684, 0.738)</td></tr><tr><td>Random</td><td>0.580 (0.550, 0.611)</td><td>0.431 (0.400, 0.465)</td><td>0.609 (0.578, 0.640)</td><td>0.405 (0.374, 0.436)</td></tr><tr><td>AV* (CMSD)</td><td>0.721 (0.704, 0.738)</td><td>0.723 (0.699, 0.746)</td><td>0.778 (0.76, 0.796)</td><td>0.685 (0.657, 0.713)</td></tr><tr><td rowspan="4">Audio</td><td>MSpecF(A*,V*) (CMSD)</td><td>0.712 (0.687, 0.737)</td><td>0.736 (0.711, 0.761)</td><td><strong>0.778</strong> (0.761, 0.794)</td><td>0.704 (0.674, 0.734)</td></tr><tr><td>MSpecF(A*,V*) (CMSD)</td><td><strong>0.743</strong> (0.724, 0.761)</td><td><strong>0.782</strong> (0.766, 0.798)</td><td><strong>0.797</strong> (0.784, 0.81)</td><td>0.716 (0.687, 0.745)</td></tr><tr><td>AV (CMAD)</td><td>0.692 (0.675, 0.710)</td><td>0.722 (0.696, 0.748)</td><td>0.774 (0.758, 0.79)</td><td>0.704 (0.676, 0.732)</td></tr><tr><td>MSpecF(A,V) (CMAD)</td><td>0.689 (0.670, 0.709)</td><td>0.726 (0.706, 0.746)</td><td>0.769 (0.749, 0.788)</td><td>0.705 (0.682, 0.728)</td></tr><tr><td rowspan="4">Audio</td><td>MSpecF(A,V,AV) (CMAD)</td><td>0.704 (0.687, 0.722)</td><td><strong>0.738</strong> (0.715, 0.761)</td><td>0.749 (0.724, 0.773)</td><td><strong>0.719</strong> (0.694, 0.744)</td></tr><tr><td>Random</td><td>0.526 (0.500, 0.552)</td><td>0.492 (0.467, 0.517)</td><td>0.609 (0.578, 0.640)</td><td>0.405 (0.374, 0.436)</td></tr><tr><td>AT* (CMSD)</td><td>0.437 (0.389, 0.484)</td><td><strong>0.788</strong> (0.759, 0.817)</td><td>0.681 (0.652, 0.71)</td><td>0.516 (0.482, 0.55)</td></tr><tr><td>MSpecF(A*,T*) (CMSD)</td><td>0.438 (0.391, 0.485)</td><td>0.779 (0.744, 0.814)</td><td>0.661 (0.626, 0.695)</td><td>0.451 (0.421, 0.48)</td></tr><tr><td rowspan="4">Text</td><td>MSpecF(A*,T*) (CMSD)</td><td><strong>0.461</strong> (0.416, 0.506)</td><td><strong>0.789</strong> (0.756, 0.823)</td><td>0.682 (0.653, 0.711)</td><td>0.497 (0.463, 0.532)</td></tr><tr><td>AT (CMAD)</td><td>0.374 (0.328, 0.421)</td><td>0.757 (0.721, 0.794)</td><td>0.697 (0.669, 0.726)</td><td><strong>0.609</strong> (0.578, 0.639)</td></tr><tr><td>MSpecF(AT) (CMAD)</td><td>0.372 (0.328, 0.416)</td><td>0.748 (0.708, 0.786)</td><td>0.675 (0.64, 0.709)</td><td>0.561 (0.52, 0.601)</td></tr><tr><td>MSpecF(AT,AT) (CMAD)</td><td>0.395 (0.352, 0.439)</td><td>0.765 (0.726, 0.804)</td><td><strong>0.699</strong> (0.67, 0.729)</td><td><strong>0.609</strong> (0.578, 0.64)</td></tr><tr><td>Random</td><td>0.320 (0.282, 0.358)</td><td>0.688 (0.651, 0.725)</td><td>0.609 (0.578, 0.640)</td><td>0.405 (0.374, 0.436)</td></tr></tbody></table>

Table 2 gathers the performance of the modality pairs: text-visual, audio-visual, and audio-text. For each modality pair, e.g., text-visual, TV* is a modality-specialized model that concatenates visual and textual tokens and processes them with a PerceiverIO model trained with CMSD. As aforementioned, MSpecF(M1*, M2*) ensembles unimodal models trained on their CMSD datasets; MSpecF(M1*, M2*, M1M2*) includes all three component models.

First, we compare modality-specific with modality-specialized models. Table 2 shows that these models perform similarly in task accuracy and trustworthiness when combining text and visual inputs, with a slight advantage for the specialized models. This is expected as CMSD of text and visual almost gathers all the samples in CMAD. Second, when comparing the fusion choices, MSpecF fusion (especially MSpecF(T*, V*, TV*)), achieves best results both in task accuracy and in trustworthiness. This indicates that fusing both unimodal and bimodal specialized models is best on both account for their modalities.

For less certain modalities such as audio and visual, specialized supervision (CMSD) improves trustworthiness, and combining all components with MSpecF(A*, V*, AV*) further boosts performance, especially positive AUC-PR. When combining two less informative modalities together, such as audio and text, we observe that specialized models with MSpecF improve trustworthiness, while task accuracy is better supported by modality-specific models. Specifically, the generic multimodal specific model (AT) outperform its MSpecF(A,T) counterpart. Similarly, the specialized AT* model achieves higher accuracy than MSpecF(A*,T*). This is expected because for less certain modality pairs, early fusion has been shown more beneficial [7]. However, with MSpecF(A*, T*, AT*) we achieve the highest trustworthiness, while the highest task accuracy is achieved with modality-specific MSpecF(A,T,AT).

Table 3 shows the performance when all three modalities are considered. The modality-specialized fusion model, MSpecF(All*), fusing all 7 component models, achieves the best task accuracy and modality trustworthiness. By aggregating predictions from all
unimodal, bimodal, and trimodal models trained with specialized supervision, MSpecF(All*) outperforms both generic multimodal models (ATV) and simpler fusion strategies, offering robust improvements in task accuracy. Fig. 4 shows the trustworthiness (modality accuracy) of the trimodal models for each modality group. We observe that MSpecF(All*) achieves consistently similar or better trustworthiness than its competitors. We assess trustworthiness using the modality misattributions metric (Sec. 4.3). Fig. 5 shows how trimodal and bimodal text-vision models are positioned based on their task accuracy and modality trustworthiness. Among bimodal models, MSpecF(T*, V*) is the most faithful, achieving the lowest misattribution error. Introducing fusion with unimodal and bimodal specialized models in MSpecF(T*, V*, TV*) results in a trade-off between attribution fidelity and task accuracy. This pattern extends to the full trimodal (audio-visual-text) setting, where MSpecF(All*), which integrates unimodal, bimodal, and trimodal specialized models, achieves a balanced compromise between task accuracy and trustworthiness.

**Answer to RQ2:** fusing fully specialized unimodal models yields best results in both task accuracy and trustworthiness (modality accuracy) for the most certain modality pair (vision and text), while when the least certain modality (audio) is involved, task accuracy is maximized by less specialized models taking both modalities as input. For three modalities, fusing unimodal, bimodal and trimodal models each trained on their respective modality-specific labels, MSpecF(All*), achieves best results in task accuracy and trustworthiness in most cases.

<figure><img src="image_12.png" /><figcaption>Figure 4: Trustworthiness (modality accuracy) of the trimodal models over each set of modalities.</figcaption></figure>

## 6 Discussion

We identify three main challenges to further adoption of our methodology: the need for modality-level annotations, the dataset size of our study and the computational complexity of our framework. Our approach is grounded in concept-based interpretability, relying on densely annotated data requiring domain expertise but crucial to assess whether models attribute predictions to the correct modalities. Although MObyGaze is moderate in size, it is comparable to other human-annotated benchmarks like UCF-Crime and MovieChat

<figure><img src="image_13.png" /><figcaption>Figure 5: Task accuracy versus trustworthiness metric of modality misattributions (defined in Section 4.3).</figcaption></figure>

<table><caption>Table 3: Trustworthiness and task accuracy of trimodal fusion strategies.</caption><thead><tr><th rowspan="2">Modality</th><th rowspan="2">Strategy</th><th colspan="2">Task accuracy</th></tr><tr><th>Pos AUC-PR</th><th>Neg AUC-PR</th></tr></thead><tbody><tr><td rowspan="4">Audio Text Visual</td><td>ATV (CMAD)</td><td>0.790 (0.771, 0.808)</td><td>0.716 (0.685, 0.747)</td></tr><tr><td>MSpecF(A*,T*,V*) (CMSD)</td><td>0.777 (0.761, 0.793)</td><td>0.703 (0.672, 0.734)</td></tr><tr><td>MSpecF(All*) (CMSD)</td><td><strong>0.802</strong> (0.790, 0.814)</td><td><strong>0.739</strong> (0.713, 0.765)</td></tr><tr><td>MSpecF(A,T,V) (CMAD)</td><td>0.770 (0.748, 0.791)</td><td>0.701 (0.673, 0.729)</td></tr><tr><td></td><td>MSpecF(All) (CMAD)</td><td><strong>0.793</strong> (0.777, 0.809)</td><td><strong>0.732</strong> (0.707, 0.758)</td></tr><tr><td></td><td>E2E</td><td>0.746 (0.725, 0.768)</td><td>0.656 (0.619, 0.693)</td></tr><tr><td></td><td>Random</td><td>0.609 (0.578, 0.640)</td><td>0.405 (0.374, 0.436)</td></tr></tbody></table>

[30, 31], and embodies the data-centric AI paradigm, where high-quality, semantically grounded annotations enable more trustworthy and interpretable models. Finally, our MSpecF framework introduces a trade-off between computational cost and trustworthiness. Combining multiple modality-specialized models increases overhead compared to a single model. However, as described in Sec. 5, we design the system to keep this cost manageable. In return, we gain in interpretability and trustworthiness, which we believe valuable in high-stakes applications.

## 7 Conclusion

We studied the trustworthiness of multimodal models and highlighted the value of using per-modality labels. We considered a complex video interpretation task of detecting character objectification in films. We have introduced our framework MSpecF to study how to flexibly specialize models thanks to per-modality labels and to fuse modular combinations of those. We have introduced two trustworthiness metrics based on these labels. We thereby showed that specific choices of model specialization and combinations can yield gains in both task accuracy and modality trustworthiness, or achieve advantageous trade-offs. Future work will study how to leverage such labels to better assess modality informativeness and information fusion. More broadly, we show that rich expert annotations for complex tasks, e.g., for social scientists, can be used to build more trustworthy models.

**Acknowledgements.** This work has been supported by the French National Research Agency through the ANR TRACTIVE project ANR-21-CE38-0012-01.
References

[1] Elisa Ancarani et al. [n. d.]. Leveraging Multimodal Explanatory Annotations for Video Interpretation with Modality Specific Dataset. arxiv

[2] Ashutosh Chaubey, Anoubh Agarwal, Sartaki Sinha Roy, Aayush Agrawal, and Susmita Ghose. 2025. ContextIQ: A Multimodal Expert-Based Video Retrieval System for Contextual Advertising. WACV (2025).

[3] Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning. arXiv:1702.08608 [stat.ML] https://arxiv.org/abs/1702.08608

[4] Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik. 2023. Towards Robust Metrics for Concept Representation Evaluation. Proc. of the AAAI Conf. on Artificial Intelligence 37, 10 (June 2023), 11791–11799. doi:10.1609/aaai.v37i10.26392

[5] C. Wah et al. 2011. The Caltech-UCSD Birds-200-2011 Dataset.

[6] D. Bose et al. 2023. MM-AU:Towards Multimodal Understanding of Advertisement Videos. In 31st ACM Int'l Conf. on Multimedia (Ottawa ON, Canada) (MM '23). ACM, NY, USA, 10 pages. doi:10.1145/3581783.3612371

[7] G.Barnum et al. 2020. On the Benefits of Early Fusion in Multimodal Representation Learning. CoRR abs/2011.07191 (2020). arXiv:2011.07191 https://arxiv.org/abs/2011.07191

[8] J. Tores et al. 2024. Visual Objectification in Films: Towards a New AI Task for Video Interpretation. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR).

[9] N. Bolin et al. [n. d.]. Expanding Language-Image Pretrained Models for General Video Recognition. In Computer Vision - ECCV 2022: 17th European Conf., Proc. Part IV (Tel Aviv, Israel). Springer-Verlag, Berlin, Heidelberg, 18 pages. doi:10.1007/978-3-031-19772-7_1

[10] R. Daneshjou et al. 2022. SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2022/file/7318b51b52078e3af28197e725f0568a-Paper-Datasets_and_Benchmarks.pdf

[11] Y. Gong et al. 2021. AST: Audio Spectrogram Transformer. CoRR abs/2104.01778 (2021). arXiv:2104.01778 https://arxiv.org/abs/2104.01778

[12] Jort F. et al. Gemmeke. [n. d.]. Audio Set: An ontology and human-labeled dataset for audio events. In 2017 IEEE Int'l Conf. on Acoustics, Speech and Signal Processing (ICASSP) (New Orleans, LA, USA). IEEE Press, 5 pages. doi:10.1109/ICASSP.2017.7952261

[13] Kristina Gligorić, Tijana Zrnic, Cinoo Lee, Emmanuel J. Candès, and Dan Jurafsky. 2025. Can Unconfident LLM Annotations Be Used for Confident Conclusions? doi:10.48550/arXiv.2408.15204 arXiv:2408.15204 [cs].

[14] Tengda Han, Max Bain, Arsha Nagrani, Gul Varol, Weidi Xie, and Andrew Zisserman. [n. d.]. AutoAD II: The Sequel - Who, When, and What in Movie Audio Description. ([n. d.]).

[15] Tengda Han, Max Bain, Arsha Nagrani, Gul Varol, Weidi Xie, and Andrew Zisserman. 2023. AutoAD: Movie Description in Context. In 2023 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). IEEE, Vancouver, BC, Canada, 18930–18940. doi:10.1109/CVPR52729.2023.01815

[16] Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, and Andrew Zisserman. 2024. AutoAD III: The Prequel - Back to the Pixels. In 2024 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). IEEE, Seattle, WA, USA, 18164–18174. doi:10.1109/CVPR52733.2024.01720

[17] Zongbo Han, Fan Yang, Junzhou Huang, Changqing Zhang, and Jianhua Yao. 2022. Multimodal Dynamics: Dynamical Fusion for Trustworthy Multimodal Classification. In 2022 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). IEEE, New Orleans, LA, USA, 20675–20685. doi:10.1109/CVPR52688.2022.02005

[18] Xuejian Huang, Tinghuai Ma, Li Jia, Yuanjian Zhang, Huan Rong, and Najla Alnabhan. 2023. An effective multimodal representation and fusion method for multimodal intent recognition. Neurocomputing 548 (Sept. 2023), 126373. doi:10.1016/j.neucom.2023.126373

[19] Andreas Jaegle et al. 2021. Perceiver IO: A General Architecture for Structured Inputs & Outputs. CoRR abs/2107.14795 (2021). arXiv:2107.14795 https://arxiv.org/abs/2107.14795

[20] Anna Kukleva, Makarand Tapaswi, and Ivan Laptev. 2020. Learning Interactions and Relationships Between Movie Characters. In 2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). IEEE, Seattle, WA, USA, 9846–9855. doi:10.1109/CVPR42600.2020.00987

[21] Ayush Kumar and Jithendra Vepa. 2020. Gated Mechanism for Attention Based Multimodal Sentiment Analysis. doi:10.48550/arXiv.2003.01043 arXiv:2003.01043 [cs].

[22] Puneet Kumar, Shreshtha Misra, Zhuhong Shao, Bin Zhu, Balasubramanian Raman, and Xiaobai Li. 2025. Multimodal Interpretable Depression Analysis Using Visual, Physiological, Audio and Textual Data. In 2025 IEEE/CVF Winter Conf. on Applications of Computer Vision (WACV). IEEE Computer Society, Los Alamitos, CA, USA, 5305–5315. doi:10.1109/WACV61041.2025.00518

[23] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proc. of the 31st Int'l Conf. on Neural Information Processing Systems (Long Beach, CA, USA) (NIPS '17). Curran Associates Inc., Red Hook, NY, USA, 6405–6416.

[24] Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2023. MultiViz: Towards Visualizing and Understanding Multimodal Models. ICLR (March 2023). doi:10.48550/arXiv.2207.00056 arXiv:2207.00056 [cs].

[25] Sijie Mai, Ying Zeng, and Haifeng Hu. 2023. Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations. IEEE Transactions on Multimedia 25 (2023), 4121–4134. doi:10.1109/TMM.2022.3171679 arXiv:2210.17444 [cs].

[26] Victor R. Martinez, Krishna Somandepalli, and Shrikanth Narayanan. 2022. Boys don't cry (or kiss or dance): A computational linguistic lens into gendered actions in film. PLOS ONE 17, 12 (Dec. 2022), e0278604. doi:10.1371/journal.pone.0278604

[27] Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence 267 (2019), 1–38.

[28] Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns 2, 11 (2021), 100336. doi:10.1016/j.patter.2021.100336

[29] Abshishek Rajora, Shubham Gupta, and Suman Kundu. 2025. Cross-Aligned Fusion For Multimodal Understanding. In 2025 IEEE/CVF Winter Conf. on Applications of Computer Vision (WACV). IEEE Computer Society, Los Alamitos, CA, USA, 5730–5740. doi:10.1109/WACV61041.2025.00559

[30] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. 2024. MovieChat: From Dense Token to Sparse Memory for Long Video Understanding. In 2024 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). IEEE, Seattle, WA, USA, 18221–18232. doi:10.1109/cvpr52733.2024.01725

[31] Waqas Sultani, Chen Chen, and Mubarak Shah. 2018. Real-World Anomaly Detection in Surveillance Videos. In 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. IEEE, Salt Lake City, UT, 6479–6488. doi:10.1109/CVPR.2018.00678

[32] Zineng Tang, Jaemin Cho, Jie Lei, and Mohit Bansal. 2023. PERCEIVER-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention. In 2023 IEEE/CVF Winter Conf. on Applications of Computer Vision (WACV). IEEE, Waikoloa, HI, USA, 4399–4409. doi:10.1109/WACV56688.2023.00439

[33] Julie Tores, Elisa Ancarani, Lucile Sassatelli, Hui-Yin Wu, Clement Bergman, Lea Andolfi, Victor Eremment, Remy Sun, Frederic Precioso, Thierry Devars, Magali Guaresi, Virginie Julliard, and Sarah Lecossais. 2025. MObyGaze: a film dataset of multimodal objectification densely annotated by experts. arXiv:2505.22084 [cs.CV] https://arxiv.org/abs/2505.22084

[34] Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang. 2020. Not only Look, But Also Listen: Learning Multimodal Violence Detection Under Weak Supervision. In Computer Vision - ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Vol. 12375. Springer Int'l Publishing, Cham, 322–339. doi:10.1007/978-3-030-58577-8_20 Series Title: Lecture Notes in Computer Science.

[35] Chejian Xu, Jiawei Zhang, Zhaorun Chen, Chulin Xie, Mintong Kang, Yujin Potter, Zhun Wang, Zhuowen Yuan, Alexander Xiong, Zidi Xiong, Chenhui Zhang, Lingzhi Yuan, Yi Zeng, Peiyang Xu, Chengquan Guo, Andy Zhou, Jeffrey Ziwei Tan, Xuandong Zhao, Francesco Pinto, Zhen Xiang, Yu Gai, Zinan Lin, Dan Hendrycks, Bo Li, and Dawn Song. 2025. MMDT: Decoding the Trustworthiness and Safety of Multimodal Foundation Models. In The Thirteenth Int'l Conf. on Learning Representations. https://openreview.net/forum?id=qlbbBSzH6n

[36] Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, and Mateja Jamnik. 2022. Concept Embedding Models. In Advances in Neural Information Processing Systems, Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (Eds.). https://openreview.net/forum?id=HXCPA2GXf

[37] Yichi Zhang, Yao Huang, Yitong Sun, Chang Liu, Zhe Zhao, Zhengwei Fang, Yifan Wang, Huanran Chen, Xiao Yang, Xingxing Wei, Hang Su, Yinpeng Dong, and Jun Zhu. 2024. MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models. In The Thirty-eight Conf. on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=5c1hh8AeHv

[38] Xiao Zheng, Chang Tang, Zhiguo Wan, Chengyu Hu, and Wei Zhang. 2023. Multi-Level Confidence Learning for Trustworthy Multimodal Classification. Proc. of the AAAI Conf. on Artificial Intelligence 37, 9 (June 2023), 11381–11389. doi:10.1609/aaai.v37i9.26346

[39] Heqing Zou, Meng Shen, Chen Chen, Yuchen Hu, Deepu Rajan, and Eng Siong Chng. 2023. UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning. In Findings of the Association for Computational Linguistics: ACL 2023, Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 659–672. doi:10.18653/v1/2023.findings-acl.41