SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haoyi Duan $^{1*}$  Yan Xia $^{1*}$  Mingze Zhou $^{1}$  Li Tang $^{1}$  Jieming Zhu $^{3}$  Zhou Zhao $^{1,2\dagger}$ $^{1}$  Zhejiang University  $^{2}$ Shanghai Artificial Intelligence Laboratory  $^{3}$ Huawei Noah's Ark Lab {haoyiduan075, xiayan.zju}@gmail.com {3200102572, tanglzju, zhaozhou}@zju.edu.cn jiemingzhu@ieee.org
In recent years, the deployment of large-scale pre-trained models in audio-visual downstream tasks has yielded remarkable outcomes. However, these models, primarily trained on single-modality unconstrained datasets, still encounter challenges in feature extraction for multi-modal tasks, leading to suboptimal performance. This limitation arises due to the introduction of irrelevant modality-specific information during encoding, which adversely affects the performance of downstream tasks. To address this challenge, this paper proposes a novel Dual-Guided Spatial-Channel-Temporal (DG-SCT) attention mechanism. This mechanism leverages audio and visual modalities as soft prompts to dynamically adjust the parameters of pre-trained models based on the current multi-modal input features. Specifically, the DG-SCT module incorporates trainable cross-modal interaction layers into pre-trained audio-visual encoders, allowing adaptive extraction of crucial information from the current modality across spatial, channel, and temporal dimensions, while preserving the frozen parameters of large-scale pre-trained models. Experimental evaluations demonstrate that our proposed model achieves state-of-the-art results across multiple downstream tasks, including AVE, AVVP, AVS, and AVQA. Furthermore, our model exhibits promising performance in challenging few-shot and zero-shot scenarios. The source code and pre-trained models are available at https://github.com/haoyi-duan/DG-SCT.
With the increasing availability of hardware resources, large-scale models [20, 4, 3] pre-trained on extensive data have achieved significant advancements in various multi-modal tasks [26, 5]. Nonetheless, since these models are primarily pre-trained on single modality, they may not be optimally suited for current multi-modal downstream tasks [33, 32, 43, 14]. As depicted in Fig 1 (a), the pre-trained model equally extracts visual features and directly passes them to downstream tasks. However, when perceiving the roaring sound of an engine, the visual region depicting a "car" should receive more attention than the region of "trees". Simultaneously, when observing the car, it is crucial to concentrate on the audio segments of the engine sound. Therefore, the encoder should not only equally extract modal-specific information from the current modality, but also highlight information related to other modalities to enhance feature fusion across diverse modalities in downstream tasks. Retraining these large models based on downstream tasks would impose an unaffordable burden [41, 22, 39], leading recent works to explore methods for fine-tuning pre-trained models on downstream tasks without full retraining, showing promising progress. However, these CLIP-based methods have primarily focused on text-image tasks [46, 45, 7, 12], while overlooking another important
Figure 1: Our Spatial and Temporal attention can focus on important regions and moments in video and emphasize critical timestamps and frequencies in audio; Channel attention enhances the representations of audio and visual features. Take visual modality, our visual features contain fine-grained, task-specific information under the guidance of audio prompts.
multi-modal scenario: audio-visual tasks. Hence, in this paper, we primarily investigate how to utilize existing large-scale models, such as CLIP [26] and Swin-Transformer [20], to adaptively adjust the encoding features with the guidance of counterpart modality when encoding audio or visual information.
The success of prompt learning in large language models (LLMs) [13, 18, 19, 30, 21] has recently sparked growing research interest in multi-modal prompt learning, as seen in works such as CoOp [46], CoCoOp [45], CLIP-Adapter [7], DenseCLIP [27], and MaPLe [12]. While these approaches enable adaptive adjustment of input features using text-based prompt templates for downstream tasks, an important question arises: Can audio or video serve as innovative prompt templates to enhance task comprehension for pre-trained models and guide adaptive feature extraction of the counterpart modality? Our findings suggest a positive answer to this question.
In this paper, we present the Dual-Guided Spatial-Channel-Temporal (DG-SCT) attention mechanism, designed to adaptively adjust feature extraction of pre-trained models based on audio-visual input. Our work is motivated by a recent work, LAVisH [17], which introduces trainable layers with shared parameters in pre-trained models to enhance fusion of audio-visual features, demonstrating promising performance on various audio-visual downstream tasks with minimal additional parameters. However, LAVisH has a few limitations. First, it relies solely on a visual encoder to encode audio, which we argue is insufficient for capturing key audio features [10, 23]. Second, it only employs cross-attention in trainable layers to introduce information from different modalities, without explicitly highlighting crucial information within the current modality. By contrast, our approach incorporates cross-modal interaction layers into audio (HTS-AT [3]) and visual (ViT [4] or Swin-T [20]) pre-trained models, leveraging different modalities as prompts to focus on special aspects of the input features that are more relevant to the counterpart modal semantics across spatial, channel, and temporal dimensions. We term our proposed approach as "prompts" to denote the guidance provided to the trainable weights in preceding layers. It emphasizes utilizing audio and video cues to guide the representation of the counterpart modalities.
As depicted in Fig. 1, unlike previous audio and visual encoders, which generate audio and visual features separately and uniformly (Fig. 1 (a)), our features contain fine-grained, task-specific information at multiple levels by leveraging the guiding characteristics of multi-modal information [37, 36]. This enables efficient implementation of a wide range of downstream tasks. Notably, unlike previous CLIP works [46, 45] that offer unidirectional prompts, our approach introduces bidirectional prompts. This means that visual and audio modalities can mutually guide each other, facilitating enhanced feature extraction from the respective modalities.
In summary, this paper makes the following contributions:
- We highlight the limitations faced by large-scale pre-trained models in audio-visual downstream tasks, which hinder their optimal performance. To overcome these, we propose to utilize audio-visual features as novel prompts to fully leverage the feature extraction capa
Figure 2: 1) Audio-visual inputs; 2) DG-SCT is injected into every layer of frozen pre-trained audio and visual encoders; 3) After feature extraction, the audio and visual features are sent to various downstream tasks; 4) Details of DG-SCT in spatial-channel-temporal attention levels.
bilities of large-scale models, enabling the effective utilization of task-specific information from different modalities.
- We introduce a novel attention mechanism named Dual-Guided Spatial-Channel-Temporal (DG-SCT), which utilizes audio and visual modalities to guide the feature extraction of their respective counterpart modalities across spatial, channel, and temporal dimensions. Notably, our approach adds only a limited number of parameters for the interaction layer, while keeping the original parameters of the large-scale pre-trained models frozen.
- Extensive experimental results on four audio-visual tasks, namely, AVE, AVVP, AVQA, and AVS, demonstrate the superiority of our model compared to state-of-the-art counterparts across various settings. Furthermore, we evaluate the performance of DG-SCT in few-shot and zero-shot scenarios on the AVE and LLP datasets, demonstrating its superiority over CLIP and several competitive CLIP-Adapters.
2.1 Audio-visual understanding
Audio-visual understanding tasks involve utilizing both audio and visual modalities to get a better perception of audio-visual scenarios [8, 22, 41, 31]. For instance, Audio-Visual Event Localization (AVE [33]) requires models to recognize joint audio-visual events. Previous works [33, 16, 35, 37, 36] use late interaction strategies to better leverage the visual and audio features encoded from modality-specific pre-trained models. Audio-Visual Video Parsing (AVVP [32]) task breaks the restriction that audio and visual signals are definitely aligned. To tackle the weakly-supervised AVVP task, previous work [32] proposes a hybrid attention network and attentive Multimodal Multiple Instance Learning (MMIL) Pooling mechanism to aggregate all features. The task of Audio-Visual Segmentation (AVS [42]) focuses on whether each pixel corresponds to the given audio so that a mask of the sounding object(s) is generated. Zhou et al. [43] use a temporal pixel-wise audio-visual interaction module to inject audio semantics as guidance for the visual segmentation process. Furthermore, the newly introduced Audio-Visual Question Answering (AVQA [14]) task requires methods that perceive both audio and visual modalities to answer human-generated questions about the audio-visual content. Li et al. propose a spatiotemporal grounding model [14] to achieve scene understanding and reasoning over audio and visual modalities.
However, most methods designed for these tasks rely on modality-specific audio and visual pretrained models, which can not utilize multi-modal cues early in the representation stage. In this paper, we propose a novel early-interaction strategy, adaptively extracting key information from the current modality across spatial-channel-temporal dimensions.
2.2 Vision-language models and prompt learning
Vision-language models have made remarkable progress since the introduction of CLIP [26], with zero-shot and few-shot ideas achieving excellent generalization abilities in many downstream tasks; Meanwhile, prompt, a concept in NLP [15, 13, 19], has achieved impressive results in various NLP domains since its introduction, as evidenced by the success of the GPT series [25, 1]. Subsequent works have attempted to combine these two and achieved better results. For example, CoOp [46] improves the CLIP model by optimizing the continuous prompts in the language branch, and CoCoOp [45] further improves the model by incorporating prompts in the video branch. However, these works only utilize prompts to guide individual branches. CLIP-adapter [7] builds on these works by proposing to use embedding of video and language to guide each other at the end of the encoder. MaPLe [12] is the first to use an adaptor to guide each other inside the encoder, integrating visual and text representations with the semantics of each other to enhance the generalization ability of the model. However, none of these works consider utilizing prompts in the audio-visual domain.
In this paper, we introduce a novel bidirectional prompt that employs audio and video cues independent of text to achieve outstanding information extraction abilities for audio-visual tasks.
In this section, more details about our proposed DG-SCT are elaborated. An overview of the proposed framework is illustrated in Fig. 2.
3.1 Representations for audio-visual modalities
Visual representation Given an input video sequence, we first sample a fixed number of RGB video frames  $\{V_t\}_{t=1}^T \in \mathbb{R}^{T \times H \times W \times 3}$ , where  $H, W$  are height and width. Following the Swin-T [20], we first split each RGB frame  $V_t$  into non-overlapping patches by a patch-splitting module with kernel size  $(P_v \times P_v)$ . Each patch is treated as a "token" and its feature is set as a concatenation of the raw pixel values. A linear embedding layer is then applied to this raw-valued feature and we can get visual features as  $\nu_t \in \mathbb{R}^{\frac{H}{P_v} \times \frac{W}{P_v} \times C_v}$ , where  $C_v$  is the number of visual channels.
Audio representation Given an input audio track, we first get an audio mel-spectrogram  $\{A_t\}_{t=1}^T$ , where  $A_t \in \mathbb{R}^{L \times F}$  with time  $L$  and frequency bins  $F$ . Following the HTS-AT [3],<sup>3</sup> the audio mel-spectrogram is cut into different patch tokens with a Patch-Embed CNN of kernel size  $(P_a \times P_a)$ . A linear embedding layer is then applied to this raw-valued feature and we can obtain audio features as  $a_t \in \mathbb{R}^{\frac{L}{P_a} \times \frac{F}{P_a} \times C_a}$ , where  $C_a$  is the number of audio channels.
3.2 Adding DG-SCT modules to frozen encoders
Now, we describe how we adjust pre-trained Swin-T and HTS-AT with our proposed DG-SCT. Every layer of the Swin-T and HTS-AT consists of three main operations: 1) multi-head attention (MHA), 2) multi-layer perceptron (MLP), and 3) our DG-SCT modules which use the intermediate layer information of audio and video as prompts to guide each other through spatial-channel-temporal dimensions. We skip the linear normalization layers in both MHA and MLP operations for brevity.
Given audio inputs  $a^{(\ell)} \in \mathbb{R}^{T \times (L^{(\ell)} \cdot F^{(\ell)}) \times C_a^{(\ell)}}$  and visual inputs  $v^{(\ell)} \in \mathbb{R}^{T \times (H^{(\ell)} \cdot W^{(\ell)}) \times C_v^{(\ell)}}$  from layer  $\ell$ , we first use a two-dimensional convolution kernel and a linear projection to make the dimensions of the audio and visual prompts consistent of their counterpart modality. Let  $v_{f}^{(\ell)} = \Omega^{\mathrm{a2v}}(a^{(\ell)}, v^{(\ell)})$  and  $a_{f}^{(\ell)} = \Omega^{\mathrm{v2a}}(v^{(\ell)}, a^{(\ell)})$  denote the operation that implements DG-SCT module, which we will describe in the next subsection. Then, the operations in each layer can be written as:
<sup>3</sup>https://github.com/RetroCirce/HTS-Audio-Transformer, an audio encoder based on SwinTransformer.
$$
v _ {y} ^ {(\ell)} = v ^ {(\ell)} + \mathrm {M H A} (v ^ {(\ell)}) + \Omega^ {\mathrm {a 2 v}} (a ^ {(\ell)}, v ^ {(\ell)}), \quad a _ {y} ^ {(\ell)} = a ^ {(\ell)} + \mathrm {M H A} (a ^ {(\ell)}) + \Omega^ {\mathrm {v 2 a}} (v ^ {(\ell)}, a ^ {(\ell)}), \tag {1}
$$
$$
v ^ {(\ell + 1)} = v _ {y} ^ {(\ell)} + \operatorname {M L P} \left(v _ {y} ^ {(\ell)}\right) + \Omega^ {\mathrm {a} 2 \mathrm {v}} \left(a _ {y} ^ {(\ell)}, v _ {y} ^ {(\ell)}\right), \quad a ^ {(\ell + 1)} = a _ {y} ^ {(\ell)} + \operatorname {M L P} \left(a _ {y} ^ {(\ell)}\right) + \Omega^ {\mathrm {v} 2 \mathrm {a}} \left(v _ {y} ^ {(\ell)}, a _ {y} ^ {(\ell)}\right). \tag {2}
$$
3.3 Dual-guided spatial-channel-temporal attention
In this subsection, we will describe how DG-SCT works in more detail. Given visual and audio features, the encoder such as Swin-Transformer pre-trained on large-scale single-modal data, will uniformly extract features from audio-visual inputs (See Fig. 1 (a)). However, in practical multi-modal scenarios, not all of this information carries equal importance. For example, as we see in Fig. 1, the region where the car appears in the visual field is evidently more crucial than the background trees. Additionally, the moment when the engine sound emerges in the audio should also be given more attention. Hence, we take advantage of the fact that audio-visual pairs can provide mutual guidance for each other, and utilize different modalities as prompts to help pre-trained models focus more on specific aspects of opposite modal inputs across spatial, channel, and temporal dimensions. Different from previous works [37, 36] which only leverage audio as guidance to extract visual features, our proposed DG-SCT module can achieve triple levels of information highlighting in two directions. We illustrate these cross-modal attention mechanisms in the following parts:
Channel-wise attention: Different channels represent different aspects of features. The introduction of channel attention can facilitate the model to ignore irrelevant features and improve the quality of representations [11]. We let the audio and video serve as mutual guidance signals and explicitly model the dependencies between channels on each other's modality. Concretely, We use  $\psi_{a}$  and  $\psi_{v}$  to denote the combinations of convolutional and linear projection in section 3.2, to encode audio and visual inputs as prompts:  $a_{t}^{\prime} = \psi_{a}(a_{t})\in \mathbb{R}^{C_{v}\times (H\cdot W)}$  and  $\nu_{t}^{\prime} = \psi_{v}(\nu_{t})\in \mathbb{R}^{C_{a}\times (L\cdot F)}$  respectively. For audio-to-visual (A2V), we use the spatial average pooling to process  $a_{t}^{\prime}$  and get  $a_{t}^{\prime \prime}\in \mathbb{R}^{C_{v}\times 1}$ , then fuse it with vision via element-wise multiplication after feeding them to projection layers  $\Theta_{a}^{c},\Theta_{v}^{c}\in \mathbb{R}^{C_{v}\times C_{v}}$  respectively, generating audio channel-guidance maps  $a_{t}^{cm} = (\Theta_{a}^{c}(a_{t}^{\prime \prime})\odot \Theta_{v}^{c}(\nu_{t}))\in \mathbb{R}^{C_{v}\times (H\cdot W)}$ . After that, we spatially squeeze the fused feature by global average pooling, denoted as  $\delta_{a}$ , Finally, a bottleneck layer  $\Phi_{a}$  follows with a sigmoid function  $\sigma$  is used, yielding channel attention maps  $M_t^{vc}$ ; Similarly, we generate V2A channel attentive maps  $M_t^{ac}$ :
$$
M _ {t} ^ {v c} = \sigma \left(\Phi_ {a} \left(\delta_ {a} \left(a _ {t} ^ {c m}\right)\right)\right) \in \mathbb {R} ^ {C _ {v} \times 1}, \quad M _ {t} ^ {a c} = \sigma \left(\Phi_ {v} \left(\delta_ {v} \left(v _ {t} ^ {c m}\right)\right)\right) \in \mathbb {R} ^ {C _ {a} \times 1}, \tag {3}
$$
where  $\nu_{t}^{cm}\in \mathbb{R}^{C_{a}\times (L\cdot F)}$  is the visual channel-guidance maps,  $\delta_v$  is spatial-wise global average pooling,  $\Phi_v$  indicates a bottleneck layer and  $\sigma$  denotes the sigmoid function.
Spatial-wise attention: Audio can improve visual feature extraction by contributing to visual attention in the spatial dimension [33]. Inspired by this, we leverage the guidance capabilities of audio and visual prompts to guide visual spatial attention and audio frequency attention, respectively. Similar to the aforementioned channel-wise attention, For A2V, we first get channel-attentive visual features  $\nu_{t}^{c} = (M_{t}^{vc} + 1) \odot \nu_{t}$ , then we element-wise multiply audio prompt and  $\nu_{t}^{c}$  after the projection of  $\Theta_{a}^{s}$  and  $\Theta_{v}^{s}$  to hidden dimension  $d = 256$ , generating audio spatial-guidance maps  $a_{t}^{sm} = (\Theta_{a}^{s}(a_{t}^{\prime}) \odot \Theta_{v}^{s}(\nu_{t}^{c})) \in \mathbb{R}^{d \times (H \cdot W)}$ . Then we use a projection layer  $\Theta^{s} \in \mathbb{R}^{1 \times d}$  with a sigmoid function  $\sigma$  to obtain spatial attention maps  $M_{t}^{vs}$ ; Similarly, we generate V2A frequency attentive maps  $M_{t}^{af}$ :
$$
M _ {t} ^ {v s} = \sigma \left(\Theta^ {s} \left(a _ {t} ^ {s m}\right)\right) \in \mathbb {R} ^ {1 \times (H \cdot W)}, \quad M _ {t} ^ {a f} = \sigma \left(\Theta^ {f} \left(v _ {t} ^ {f m}\right)\right) \in \mathbb {R} ^ {1 \times (L \cdot F)}, \tag {4}
$$
where  $\nu_{t}^{f m}$  denotes visual frequency-guidance maps,  $\Theta^f\in \mathbb{R}^{1\times d}$  is a projection layer.
Temporal-gated attention: Given an audio, significant time segments (e.g., "engine sound") should be emphasized, while background information (e.g., "silence") should be attenuated. The same holds for the visual information as well [36]. Inspired by this, we add temporal-gated attention in the final layer. For A2V, we first feed the frequency-channel attentive audio features  $\{a_t^{cf}\}_{t=1}^T$  through an RNN to capture temporal information better and then pass it through a projection layer with
sigmoid function to obtain the temporal attention gates  $G^{v} \in \mathbb{R}^{T \times 1}$ ; Similarly, for V2A, we feed the spatial-channel attentive visual features to generate  $G^{a} \in \mathbb{R}^{T \times 1}$ :
$$
G ^ {v} = \sigma \left(\Theta_ {a} ^ {t} \left(\operatorname {R N N} \left(\left\{a _ {t} ^ {c f} \right\} _ {t = 1} ^ {T}\right)\right)\right), \quad G ^ {a} = \sigma \left(\Theta_ {v} ^ {t} \left(\operatorname {R N N} \left(\left\{v _ {t} ^ {c s} \right\} _ {t = 1} ^ {T}\right)\right)\right). \tag {5}
$$
Summary: We combine the abovementioned three attention mechanisms together to form our DG-SCT module. Given audio features  $a_{t} \in \mathbb{R}^{C_{a} \times (L \cdot F)}$  and visual features  $\nu_{t} \in \mathbb{R}^{C_{v} \times (H \cdot W)}$ , DG-SCT first generates channel-wise attention maps  $M_{t}^{vc}$  and  $M_{t}^{ac}$  to let audio and video adaptively emphasize informative features of the corresponding modality. It then lets audio pay attention to the important sounding regions to produce spatial-wise attention maps  $M_{t}^{vs}$  and lets video pay attention to the important frequency regions to generate frequency-wise attention maps  $M_{t}^{af}$ , thus the yielding of the spatial-channel attentive visual features  $\nu_{t}^{cs}$  and the frequency-channel attentive audio features  $a_{t}^{cf}$  can be summarized as:
$$
v _ {t} ^ {c s} = \left(\alpha \cdot M _ {t} ^ {v c} + \beta \cdot M _ {t} ^ {v s} + 1\right) \odot v _ {t}, \quad a _ {t} ^ {c f} = \left(\alpha \cdot M _ {t} ^ {a c} + \beta \cdot M _ {t} ^ {a f} + 1\right) \odot a _ {t}, \tag {6}
$$
Then we generate two temporal attention gates  $G^{v} \in \mathbb{R}^{T \times 1}$  and  $G^{a} \in \mathbb{R}^{T \times 1}$  for  $\{v_{t}^{cs}\}_{t=1}^{T}$  and  $\{a_{t}^{cf}\}_{t=1}^{T}$ , respectively, thus the final outputs of  $\Omega^{a2v}$  and  $\Omega^{v2a}$  mentioned in section 3.2 are:
$$
\left\{\nu_ {t} ^ {c s t} \right\} _ {t = 1} ^ {T} = (\gamma \cdot G ^ {v} + 1) \odot \left\{\nu_ {t} ^ {c s} \right\} _ {t = 1} ^ {T}, \quad \left\{a _ {t} ^ {c f t} \right\} _ {t = 1} ^ {T} = (\gamma \cdot G ^ {a} + 1) \odot \left\{a _ {t} ^ {c f} \right\} _ {t = 1} ^ {T}, \tag {7}
$$
Where  $\alpha$ ,  $\beta$ , and  $\gamma$  are hyperparameters. Consequently, this approach yields high-quality, fine-grained audio-visual representations, significantly improving the performance of subsequent tasks in the audio-visual domain.
4.1 Tasks and datasets
Audio-visual event localization (AVE) focuses on recognizing an audio-visual event that is both visible and audible throughout multiple time segments in a video. We evaluate the AVE [33] dataset; Audio-visual video parsing (AVVP) aims to parse a video into temporal event segments and label them as either audible, visible, or both. We evaluate our method for weakly-supervised AVVP task on the LLP dataset [32]; The goal of Audio-visual segmentation (AVS) is to output a pixel-level map of the object(s) that produce sound at the image frame. We evaluate on AVSBench [43]; Audio-visual question answering (AVQA) aims to answer questions based on the associations between objects and sounds. We conduct our experiments on the MUSIC-AVQA dataset [14].
Meanwhile, We propose Audio-visual few-shot/zero-shot tasks on AVE [33] and LLP [32] datasets. We evaluate AVE and classification tasks on AVE dataset and classification task on LLP dataset. More details about tasks and datasets will be illustrated in Appendix.