# Leveraging Multimodal Explanatory Annotations For Video Interpretation With Modality Specific Dataset

Elisa Ancarani, Julie Tores, Lucile Sassatelli, Rémy Sun, Hui-Yin Wu,

Frédéric Precioso

► To cite this version:

Elisa Ancarani, Julie Tores, Lucile Sassatelli, Rémy Sun, Hui-Yin Wu, et al. Leveraging Multi-modal Explanatory Annotations For Video Interpretation With Modality Specific Dataset. 2026. [hal-05462522]

HAL Id: hal-05462522

https://hal.science/hal-05462522v1

Preprint submitted on 16 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
arXiv:2504.11232v1 [cs.CV] 15 Apr 2025

# LEVERAGING MULTIMODAL EXPLANATORY ANNOTATIONS FOR VIDEO INTERPRETATION WITH MODALITY SPECIFIC DATASET

Elisa Ancarani¹, Julie Tores¹, Lucile Sassatelli¹, Rémy Sun¹,², Hui-Yin Wu², Frédéric Precioso¹,²

¹Université Côte d'Azur, CNRS, I3S, France

²Université Côte d'Azur, Inria, France

## ABSTRACT

We examine the impact of concept-informed supervision on multimodal video interpretation models using MOByGaze, a dataset containing human-annotated explanatory concepts. We introduce Concept Modality Specific Datasets (CMSDs), which consist of data subsets categorized by the modality (visual, textual, or audio) of annotated concepts. Models trained on CMSDs outperform those using traditional legacy training in both early and late fusion approaches. Notably, this approach enables late fusion models to achieve performance close to that of early fusion models. These findings underscore the importance of modality-specific annotations in developing robust, self-explainable video models and contribute to advancing interpretable multimodal learning in complex video analysis.

**Index Terms**— video interpretation task, explanatory concept annotation, multimodal fusion

## 1. INTRODUCTION

In analysis of visual data, different types of tasks require different levels of human expertise. For example, object recognition or semantic segmentation are often less challenging for a human than assessing whether a skin lesion is malignant [1] or a meme is offensive [2].

For such complex image analysis tasks, which we call image interpretation, it is essential to have explainable models where the patterns behind decisions remain accessible and auditable to human experts.

Towards this goal, some image analysis datasets have been created with human experts providing a label for a complex interpretive task, along with the explanatory factors for their labeling decision. These are referred to as concepts. For example, CUB is an image dataset of 200 labeled bird species and annotated with 312 concepts attributes by experts [3], while SkinCon is an dataset of skin lesion images labeled as malignant or benign along with 48 explanatory concepts. Such concept annotations as ground-truth explanations allow both to assess the explainability of models and design more efficient explainable models.

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /><figcaption>**Fig. 1:** Two positive samples for character objectification from the MOByGaze dataset. The concepts annotated indicate that objectification is due to the visual modality only in (a), and to the textual modality only in (b). For sample (a) (resp. (b)), the model fed with text only (resp. vision only) and trained with the Modality Agnostic Dataset wrongly detects objectification, while it correctly classifies the sample as negative when trained on the text-specific dataset T-CMSD (resp. vision-specific dataset V-CMSD).</figcaption></figure>

In video, datasets with concept annotations are rare owing to the tedious annotation process. Tores et al. recently introduced a dataset for a challenging video interpretation task: detecting whether a character in a movie clip is objectified—that is, portrayed as an object of desire or service rather than as an active subject [4]. A prominent aspect of this video dataset is the dense annotation of 20 movies by experts with concept-based explanations for their judgments.

For such interpretive tasks in image or video, models must allow analysis of each modality's influence on the final decision to be deemed trustworthy and fulfill their purpose, such as providing objective measures of bias in visual media.

In this article, we consider video interpretation and investigate whether and how annotation of explanations as concepts can benefit multimodal models, to eventually produce better explainable models. We specifically focus on the power of models to discriminate between what modalities explain the final label, as exemplified in Fig. 1. To do so, we consider two types of multimodal fusion, early and late fusion.

<footnote>This work was supported by the French National Research Agency ANR TRACTIVE project ANR-21-CE38-0012-01.</footnote>
In video, several recent works address visual tasks informed by other modalities, like automatic auto- description [5], video-question answering [6] or social message detection [7]. These works build on early fusion of modalities, where tokens of different modal- ities are fed to a common Transformer network. This type of fusion however post-hoc explainability meth- ods, such as saliency maps of the input data, whose brittleness has been recurrently shown [8]. Late fusion on the other hand is required in self-explainable neural network models (e.g., [9, 10, 11, 12, 13]), where the production of the final decision (based on the impor- tance of meaningful higher-level concepts) is directly interpretable by humans.

**Our contributions are:**

* We consider a recent video dataset with human anno- tations of multimodal (visual, textual, audio) explana- tory concepts for a task of video interpretation. We design an assessment strategy with so-called Concept Modality Specific Datasets (CMSDs), which are sorted subsets based on annotated concept modalities. We assess the discriminative power of multimodal mod- els when trained with such modality-informed supervi- sion, in comparison with legacy training with Concept Modality Agnostic Dataset (CMAD).
* We show that training with CMSD significantly im- proves the performance of both late and early fusion models.
* We show that modality-informed supervision allow late fusion models to perform as well as early fusion, thereby showing the importance of modality-detailed annotation to enable high-performing self-explainable models for video interpretation tasks.

## 2. RELATED WORKS

Our research focuses on video interpretation at the in- tersection of two areas: (1) concept-annotated datasets and (2) multimodal fusion. As we show below, we are the first to explore concept annotations in multimodal training to enhance model performance.

**Concept-annotated Datasets** Many image and video datasets are anotated (often partially automatically) with multiple semantic aspects. For example, 3MAS- SIV is a multilingual, multimodal dataset of short social media videos annotated with concepts, affec- tive states, media types, and audio language. How- ever in this article, we consider concept annotations as ground-truth explanations provided by humans on their labeling decision for an end interpretive task. Example of such image datasets are CUB [3] and SkinCon [1] introduced in Sec. 1. Video interpreta- tion tasks are rarer. A recent dataset is MM-AU [7], of 8399 advertisement videos annotated by humans for three interpretive tasks: topic categorization, per- ceived tone transition, and social message detection. However, MM-AU does not provide annotations spec- ifying whether the end label depends on some or all

modalities (visual, textual, or audio). More recently, Tores et al. introduced the ObyGaze dataset [4], where movie clips have been densely annotated by experts for the presence of character objectification. As this is a high-level interpretive construct, annotators were required to provide explanations for their judgment. For each label of objectification presence or absence, they indicate wheteher it is due to one or several of the 8 visual concepts (such as body, clothing, type of shot, etc.). In our work, we consider the multimodal extension of this dataset, called MObyGaze, which further described in Sec. 3. It features 20 films anno- tated with 11 concepts spanning all three modalities (visual, textual and audio). We therefore explore how to best learn from this unique dataset tying multimodal explanations to an interpretive task label.

**Multimodal Fusion** Fusing information from multiple modalities is a frequent challenge in numerous tasks (such as violence detection in multimedia content [14] or intent recognition [15]). We can roughly classify the approaches into early and late fusion. Early fusion combines raw data from different modalities at the input level, while late fusion processes each modality independently before combining outputs. Early fu- sion has demonstrated increased robustness to noisy inputs, as information from one modality can help mitigate noise in another [16]. It also effectively ex- ploits cross-modal correlations when the data is well- aligned [17]. Early fusion is however not amenable to explainability approaches beyond post-hoc meth- ods. late fusion, on the other hand, preserves distinct modalities until or close to the final decision (in final layers). This allows more reliable interpretations of each modality’s contribution to the final outcome (see, e.g., SENN [9]), but comes at the cost of lower re- silience to noise—particularly when a modality is less informative or introduces interference [16].

When introducing the above-mentioned MM-AU dataset, Gupta et al. explore both unimodal and mul- timodal baselines, combining different modality pairs using a mix of early and late fusion. Specifically, they apply early fusion to the vision-text and vision- audio modality pairs, while performing late fusion by combining class probabilities at inference. Given the similarity of their tasks to objectification tasks and the availability of concept annotations, we build on their models to investigate early fusion, focusing exclusively on early fusion of modality pairs.

## 3. METHOD

We present here our method to investigate how con- cept annotations can benefit multimodal architectures for video interpretation, and whether early and late fusion benefit differently.

**Case study:** As introduced in Sec. 1, we consider detecting objectification in videos, which occurs when
a character is portrayed as an object of desire or service rather than a subject of action (often referred to as the “male gaze”). This is a task recently introduced by Tores et al. [4] with the ObyGaze12 dataset. Here we consider its multimodal extension, the MObyGaze dataset available online<fnref n="1" />. Objectification is produced by a combination of cinematic techniques (camera position, angle, movement), iconographic choices (visible body parts, clothing, looks, etc.), narrative (speech) and auditory components (voice and soundtrack). MObyGaze comprises 5,783 video clips from 20 movies densely annotated by human experts with objectification levels: Easy Negative (EN) where no objectifying factor is present, Hard Negative (HN) where at least one objectifying factor/concept is present, but not sufficient to fully perceive the character as an object, and Sure (S) where objectifying concepts are sufficiently present to produce the perception of objectification. In this article, as we focus on identifying modalities involved in the production of objectification, we consider HN and S samples as positive, as further described below.

Fig. 2 illustrates the distribution of objectification labels across movies. Fig. 3 showcases the distribution of concept modalities for each movie. Vision emerges as the dominant modality. This highlights this dataset’s particular suitability for visual interpretation tasks.

<figure><img src="image_4.png" /><figcaption>**Fig. 2:** Distribution of objectifying labels by frequency and duration in the MObyGaze dataset.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Fig. 3:** Normalized distribution of visual, text, and audio concepts across movies.
**Assessment Strategy:** We focus on modality attribu-</figcaption></figure>

tion errors, where objectification is attributed to an incorrect source compared to the ground truth annotations. A clip is labeled as positive (HN or S) if it contains at least one objectifying concept in any of the visual, audio, or textual modalities. For example, as shown in Fig. 1, in case (a), a model fed with text only may wrongly classify a sample as objectifying, even though concept annotations indicate that objectification is only produced by the visual modality. Such discrepancies reveal the model exploitation of undesired biases in the data, compromising trustworthiness and explainability. To probe model weaknesses and leverage concept annotations, we define two types of datasets from MObyGaze:

* **Concept Modality Agnostic Dataset (CMAD):** This is the standard MObyGaze split with all positive clips (HN and S) based on speech, text, and/or audio, regardless of input modality, while negative clips consist only of EN clips. This follows the CMAD approach, which considers all objectifying samples without distinguishing the modality of objectification.

* **Concept Modality Specific Dataset (CMSD):** We propose CMSD as a concept-based supervision dataset. The positive set includes HN and S clips with at least one annotated concept of the model input modality. The negative set consists of EN clips together with the clips that do not meet the positive criteria. As shown in Fig. 4, CMSD is derived from CMAD by filtering positive samples to retain only modality-relevant concepts.

As outlined in Table 1, all models are evaluated on their respective CMSD test set. However, they can be trained separately using either CMAD or the CMSD specific to their input modality. For example, Vision-only models are evaluated on Vision-CMSD, which contains as positives the HN and S of the CMAD that contains at least one concept related to the vision modality. These models can however be trained either on CMAD or on Vision-CMSD. This strategy allows for a comprehensive assessment of the models’ ability to disambiguate the source of objectification across modalities, depending on whether supervision leverages concept information.

We exclude Text-Vision pairing from our analysis as it represents almost the entire dataset. Therefore, it would limit our ability to effectively discriminate between results of CMAD and CMSD models.

<table><caption>**Table 1:** Training and testing sets for models with different input modalities.</caption><thead><tr><td>Input Modalities</td><td>Training Sets</td><td>Test Sets</td></tr></thead><tbody><tr><td>Vision</td><td>CMAD or V-CMSD</td><td>V-CMSD</td></tr><tr><td>Audio</td><td>CMAD or A-CMSD</td><td>A-CMSD</td></tr><tr><td>Text</td><td>CMAD or T-CMSD</td><td>T-CMSD</td></tr><tr><td>Audio-Vision</td><td>CMAD or AV-CMSD</td><td>AV-CMSD</td></tr><tr><td>Audio-Text</td><td>CMAD or AT-CMSD</td><td>AT-CMSD</td></tr><tr><td>Vision-Audio-Text</td><td>CMAD or AVT-CMSD</td><td>AVT-CMSD</td></tr></tbody></table>

<footnote n="1">¹https://anonymous.4open.science/r/MObyGaze-F600/</footnote>
<figure><img src="image_6.png" /><figcaption>**Fig. 4:** The Concept Modality Agnostic Dataset (CMAD) contains all positive samples (Hard Negative and Sure) across modalities, while the Concept Modality Specific Dataset (CMSD) filters these samples to include only those with modality-specific concepts.</figcaption></figure>

**Models:** To investigate how different model designs can benefit from CMSD training (thanks to concept information) compared with legacy CMAD training, we consider early and late fusion of modalities. First, we consider **Unimodal Models** fed with a single modality, processing either vision, audio, or text. Each such model is a set of Transformer layers fed with modality tokens obtained from pre-trained models (see Sec. 4). Unimodal models allow to assess the benefit of the CMSD strategy regardless of any multimodal fusion choice.

Second, we consider **Mulitmodal Models** to study the interaction between fine-grained supervision and fusion. We focus on two main strategies: early and late fusion. Our choice for **early fusion** is shown in Fig. 5a. We build on the transformer-based framework from [7]. Token projections of paired modalities are processed jointly within two shared transformer models: one combines vision and text tokens, the other vision and audio tokens. The two models are trained separately. For the early fusion model with three input modalities, the output probabilities of each pair model are then max-fused at inference, as depicted.

For late fusion, we consider three modality-specific models, each fed with a single modality as shown in Fig. 5b. They are trained independently, and their respective output probabilities are max-fused at inference time, being it for the case of the late fusion model with three input modalities depicted in Fig. 5b, or when only two modalities are considered.

## 4. EXPERIMENTS

We investigate experimentally ((Research Question - RQ1) whether training on our CMSD dataset splitsences model performances, and (RQ2) we look into whether late and early fusion schemes benefit differently from CMSD training.

**Dataset:** We perform experiments on the MObyGaze dataset discussed in previous sections.

**Model settings:** To preprocess the visual modality, we

<figure><img src="image_7.png" /><figcaption>(a) Early Fusion Model</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) Late Fusion Model</figcaption></figure>

<caption>**Fig. 5:** Comparison of Early and Late Fusion Models considering all three modalities. Tokens are: Audio (orange), Video (blue), and Text (green).</caption>

leverage X-CLIP [18], an extension of CLIP [19] designed for videos. The pre-trained X-CLIP model is kept frozen, and we extract a feature vector from each 16-frame window with a stride of 16, for each annotated video segment. For textual modality, we extract textual cues from subtitles and encode them using the pretrained BERT model (bert-base-uncased) available in the Hugging Face library [20, 21].

For audio, we segment movie audio tracks into two-second chunks, and process them using the AST feature extractor [22], pre-trained on AudioSet [23]. For training, validation, and testing, we use a five-fold split across 20 movies. The training is done with random oversampling on the minority class. We use a validation set to stop the training with early stopping with patience of 15. The backbone architecture for all the unimodal and multimodal models is based on PerceiverIO [24], chosen for its ability to scale efficiently while handling high-dimensional inputs from multiple modalities. The model is configured with a depth of 2 layers, 32 latent variables, and a latent dimension of 128. It employs 1 cross-attention head and 8 latent self-attention heads, each with a dimensionality of 32. All models, have a comparable number of trainable parameters. In the early fusion method, each of the two model contains approximately 1 million parameters. Similarly, the three independent models in the late fusion also have about 1 million parameters each.

**Results:** To evaluate performance while accounting for class imbalance, we report AUC-PR scores. Our analysis focuses on three key dimensions:

* **Training:** Concept Modality-Agnostic Dataset (CMAD) vs. Modality-Specific Dataset (CMSD).

* **Modalities:** Vision, Audio, Text, Audio-Vision, Audio-Text, Vision-Audio-Text.

* **Types of fusion:** Early vs. Late fusion.

Fig. 6-8 illustrate these dimensions, with each figure
representing different modality sets. Color intensity indicates the training strategy (darker for CMSD, lighter for CMAD). Error bars indicate confidence intervals.

Fig. 6 represents the performance of a Vision-only model (left) and Text-only model (right), when trained without CMAD or with CMSD (concept-based supervision). For both single modalities, we observe significant improvement with concept-based supervision: a 6.2% increase in AUC-PR for Vision-only and a 63% increase for Text-only. The substantial improvement in the Text-only model stems from the rarer occurrence of samples annotated with textual concepts compared to visual ones. This limited representation makes discriminating positive samples more challenging, yielding greater benefits from concept-informed supervision. Let us specify that, while we report AUC-PR for conciseness, we confirm that the increase in AUC-PR is mostly due to the increase in precision. This confirms the interest of concept-based supervision to mitigate the modality-attribution error exemplified in Fig. 1.

Fig. 7 shows the performance of models using paired modalities as input: Vision-Audio (left) and Text-Audio (right). We compare early and late fusion approaches for each pair. Concept-based supervision significantly improves performance for both modality pairs and fusion types. Text-Audio models benefit more from concept-based supervision, with a 30.0% increase in early fusion and 25.6% increase in late fusion. In contrast, Vision-Audio models demonstrate more modest improvements, with increases of 3.72% in early fusion and 3.92% in late fusion. This pattern mirrors the observations in Fig. 6.

Fig 8 reveals the impact of concept-based supervision on Early and Late fusion models combining all three modalities (vision, text, audio). In this scenario, CMSD equals CMAD, creating a less challenging discrimination task as the test set contains all samples from all modalities.

Two key observations emerge from this analysis. First, early fusion demonstrates greater resilience to the absence of concept-based supervision compared to late fusion. This stems from early fusion's ability to learn multimodal features, which enhances its robustness across modalities. Second, concept-based supervision significantly narrows the performance gap between late fusion and early fusion models. Without concept-based supervision, late fusion performs 7.25% worse than early fusion. However, with concept-based supervision, this gap reduces to just 3.30%, with overlapping confidence intervals.

These findings answer our initial research questions. RQ1: Concept-based supervision improves performance by reducing attribution errors across all models and modality sets. RQ2: With all three modalities combined, late fusion benefits most from concept-based supervision, closing the performance gap with early fusion.

<figure><img src="image_9.png" /><figcaption>Fig. 6: Left: Video as single input modality. Right: Text as single input modality.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 7: Left: Video and audio as input modalities to early and late fusion models. Right: Text and audio as input modalities to early and late fusion models.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 8: Concept-based supervision for early and late fusion of all three video, audio, and text modalities.</figcaption></figure>

## 5. CONCLUSION

In this work, we address the challenge of improving model performance in multimodal video interpretation through concept-informed supervision. We demonstrated that the integration of concepts - which serve as explanation for the final label - into the training process improves the models performance while inherently creates a more interpretable framework. Leveraging MOByGaze, a recent video dataset with human-annotated multimodal concepts, we design an assessment strategy using Concept Modality Specific Datasets (CMSDs). These datasets are created by categorizing video samples based on specific modalities, such as visual, textual, or audio, that contain explanatory concepts for the given task. This categorization enables the evaluation of the discriminative power of multimodal models. Our experiments reveal that training with CMSDs significantly improves the performance of both late and early fusion models. Notably, this modality-informed supervision enables late fusion models to achieve comparable performance to early fusion models, highlighting the importance of detailed annotations in developing robust and self-explainable video interpretation model.
## 6. REFERENCES

[1] R. Daneshjou et al., “Skincon: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis,” in *Advances in Neural Information Processing Systems*, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds. 2022, vol. 35, Curran Associates, Inc.

[2] K. Douwe et al., “The hateful memes challenge: Detecting hate speech in multimodal memes,” in *Advances in Neural Information Processing Systems*, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, Eds. 2020, vol. 33, Curran Associates, Inc.

[3] C. Wah et al., “The caltech-ucsd birds-200-2011 dataset,” 2011.

[4] J. Tores et al., “Visual objectification in films: Towards a new ai task for video interpretation,” in *IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, June 2024.

[5] Tengda Han et al., “AutoAD II: The Sequel - who, when, and what in movie audio description,” in *ICCV*, 2023.

[6] Li Yanwei et al., “Llama-vid: An image is worth 2 tokens in large language models,” in *Computer Vision – ECCV 2024, Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, Eds.*, Cham, 2025, Springer Nature Switzerland.

[7] D. Bose et al., “Mm-au: towards multimodal understanding of advertisement videos,” in *31st ACM International Conference on Multimedia*, New York, NY, USA, 2023, MM ’23, Association for Computing Machinery.

[8] C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” *Nature Machine Intelligence*, vol. 1, pp. 206–215, 2018.

[9] Alvarez-Melis et al., “Towards robust interpretability with self-explaining neural networks,” in *Proceedings of the 32nd International Conference on Neural Information Processing Systems*, Red Hook, NY, USA, 2018, NIPS’18, Curran Associates Inc.

[10] Zhi Chen et al., “Concept whitening for interpretable image recognition,” *Nature Machine Intelligence*, vol. 2, no. 12, pp. 772–782, 2020.

[11] Yao Ming et al., “Interpretable and steerable sequence learning via prototypes,” in *Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining*, New York, NY, USA, 2019, KDD ’19, Association for Computing Machinery.

[12] Mert Yuksekgonul et al., “Post-hoc concept bottleneck models,” in *The Eleventh International Conference on Learning Representations*, 2023.

[13] M. et al. Espinosa Zarlenga, “Concept embedding models: Beyond the accuracy-explainability trade-off,” *Advances in Neural Information Processing Systems*, vol. 35, 2022.

[14] W. Peng et al., “Not only look, but also listen: Learning multimodal violence detection under weak supervision,” in *Computer Vision – ECCV 2020, Cham*, Springer International Publishing.

[15] X. Huang et al., “An effective multimodal representation and fusion method for multimodal intent recognition,” *Neurocomputing*, vol. 548, 2023.

[16] G.Barnum et al., “On the benefits of early fusion in multimodal representation learning,” *CoRR*, vol. abs/2011.07191, 2020.

[17] K. Gadzicki et al., “Early vs late fusion in multimodal convolutional neural networks,” in *2020 IEEE 23rd International Conference on Information Fusion (FU-SION)*, 2020, pp. 1–6.

[18] N. Bolin et al., “Expanding language-image pretrained models for general video recognition,” in *Computer Vision – ECCV 2022: 17th European Conference, Proceedings, Part IV*, Berlin, Heidelberg, Springer-Verlag.

[19] A. Radford et al., “Learning transferable visual models from natural language supervision,” in *International conference on machine learning*. PMLR, 2021, pp. 8748–8763.

[20] J. Devlin et al., “BERT: pre-training of deep bidirectional transformers for language understanding,” *CoRR*, vol. abs/1810.04805.

[21] T. Wolf et al., “Huggingface’s transformers: State-of-the-art natural language processing,” *CoRR*, vol. abs/1910.03771, 2019.

[22] Y. Gong et al., “AST: audio spectrogram transformer,” *CoRR*, vol. abs/2104.01778, 2021.

[23] Jort F. et al. Gemmeke, “Audio set: An ontology and human-labeled dataset for audio events,” in *2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE Press.

[24] A. Jaegle et al., “Perceiver IO: A general architecture for structured inputs & outputs,” *CoRR*, vol. abs/2107.14795, 2021.