CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization

Authors: xiang  he
xiangxi  liu
yang  li
dongcheng  zhao
guobin  shen
qingqun  kong
xin  yang
yi  zeng
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 11:29:03

Abstract: The audio-visual event localization task requires identifying concurrent visual and auditory events from unconstrained videos within a model, locating them, and classifying their category. The efficient extraction and integration of audio and visual modal information have always been challenging in this field. In this paper, we introduce CACE-Net, which differs from most existing methods that solely use audio signals to guide visual information. We propose an audio-visual co-guidance attention mechanism that allows for adaptive bi-directional cross-modal attentional guidance between audio and visual clues, thus reducing inconsistencies between modalities. Moreover, we have observed that existing methods have difficulty distinguishing between similar background and event and lack the fine-grained features for event classification. Consequently, we employ background-event contrast enhancement to increase the discrimination of fused features and fine-tuned pre-trained model to extract more discernible features from complex multimodal inputs. Experiments on the AVE dataset demonstrate that CACE-Net sets a new benchmark in the audio-visual event localization task, proving the effectiveness of our proposed methods in handling complex multimodal learning and event localization in unconstrained videos. Code is available at https://github.com/Brain-Cog-Lab/CACE-Net.

Main paper: 
 introduction: 
 Our brains perceive concepts and guide behavior by integrating clues from multiple senses, a process that relies on complex and efficient information processing mechanisms [3, 22] . This multisensory integration not only improves the robustness of perception but also its efficiency. Just as the brain enhances comprehension and decision-making through clues from multiple senses, multimodal learning also integrates information across different sensory modalities to achieve a more accurate and comprehensive understanding of the context. Especially in complex tasks, multimodal contextual comprehension is crucial for the current task and future predictions [25] . The task of audio-visual event localization [28] is a specific demonstration of this concept, which requires the identification of concurrent visual and auditory events from unconstrained videos rich in visual images and audio signals. As shown in Figure 1 , for an unconstrained video, the audio-visual event exists only if the video's audio and visual information is matched in the video segment; all other cases are considered as background. Compared to action recognition that relies solely on visual information, audiovisual event localization requires a higher level of context analysis and comprehension due to the interference of asynchronous visual and audio signals, thereby increasing the complexity of recognition.In recent years, researchers have proposed various methods to improve the performance of this task. For example, AVEL [28] utilizes a cross-modal attention mechanism to guide the processing of complex visual information with audio signals; CMRAN [31] further explores information within and across modalities. Despite significant progress, we argue that the task still faces three key challenges: 1) Interference in audio signals: the presence of noise or weak signals in audio can interfere with the process of guiding visual modality and with the generation of fused features. 2) Difficulty in distinguishing between background and event: The logic underlying event classification and distinction between background and event differs. The definition of "mismatch between audio and visual clues" results in a lack of distinct features for background, leading to challenges. 3) The need for fine-grained features: The main reason for misclassification of different events is the lack of fine-grained features, especially when events resemble one another.To mitigate the above challenges, method design should follow the following principles: 1) Balance multimodal input: Reduce noise interference in the audio signal to avoid its excessive impact on feature fusion and event localization. 2) Fine-grained feature representation: Precisely capture background details and discriminate the subtle differences between the event and the background from the complex multimodal inputs. 3) Stable and efficient feature encoding: Encoders need to have strong generalization capability for feature extraction to achieve event classification more accurately.In this paper, we present an audio-visual co-guidance attention mechanism. Unlike the previous approach of using only audio signals to guide visual modality, visual information is also used to guide audio modality. The co-guidance mechanism provides different information for event localization based on visual and audio modalities, and it allows the visual and audio modalities to obtain guidance signals from each other. The audio-visual coattention mechanism reduces the impact of misleading information on audio-visual event localization by reducing the inconsistency of inter-modal information. Additionally, to address the challenge of distinguishing between background and event, we integrate a contrastive learning strategy into our model. This strategy involves intentionally perturbing features extracted by the model to simulate disturbance in scenes, which enhances the model's comprehension of background and event features, and focuses the learning process on distinguishing event from background containing information relevant to event more effectively. Finally, we fine-tune efficient visual and audio encoders specifically to extract more refined and discriminative features from complex multimodal inputs, thus enhancing the model's capability for generalized feature extraction. These three coordinated strategies are combined to propose a novel audio-visual event localization model that surpasses existing stateof-the-art methods in handling audio-visual event localization tasks.Our contribution can be summarized as follows:• Co-guidance attention: we introduce a novel audio-visual coguidance attention mechanism, which effectively reduces the inconsistency of inter-modal information and the influence of misleading information on event localization through bidirectional guidance between visual and audio modalities. • Background and event contrast enhancement: we use a contrastive learning strategy by randomly perturbing the fused features, e.g., adding noise, to enhance the contrast between event and background, and deepen the model's comprehension of background and event features. • A more efficient feature extractor: we choose more advanced visual and audio encoders, and perform targeted fine-tuning in the audio-visual event localization task to extract finegrained features from complex multimodal inputs. 
related work: 
 We begin with an introduction to the work on audio-visual learning, followed by a discussion of the attention mechanism and fine-tuning of the feature extractor on audio-visual event localization.Audio-visual learning. The main goal of audio-visual learning is to mine the relationship between audio and visual modalities, in which feature fusion is the core research direction. The research ideas are mainly divided into two categories, one is to use supervisory signals to complete the fusion of visual and audio information [12, 19] , or to use one modality as a supervisory signal to drive the information mining of the other modality [1, 23] . The other focuses on cross-modal learning using unsupervised methods when there is a known correspondence between audio and visual modalities [2, 13, 20] . Development in the field of audio-visual learning has also led to an increasing number of tasks, including video sound separation [5] , video sound source localization [14] , action recognition [6] , and audio-visual event localization [18, 28] , etc. This paper focuses on the audio-visual event localization task.Attention mechanisms. The attention mechanism on audiovisual event localization task was first proposed by [28] , which explored the spatial correspondence of audio and visual modalities by using audio modality to guide the visual modality to implement a cross-modal attention mechanism. Later Wu et al. [29] proposed a dual-attention module to obtain the similarity of the two modalities. In addition, Xu et al. [31] proposed the module to guide the visual modality using audio information at spatial and channel levels to further explore the relationship between audio and visual modalities. However, the above work ignores the help of visual modality in guiding audio signals further. Feng et al. [4] use bi-directional guidance for closer audio-visual correlation. Our work improves on the above cross-modal attention mechanisms and can further reduce the inconsistency of information between audio and visual modalities. Unlike the introduction of more complex network module in CSS [4] , we simply use the basic feature information for visual and audio bi-directional guidance to minimize the interference caused by noise in audio and visual signal when fusing features.Fine-tuning of the feature extractor. In order to improve the suitability of features extracted by audio and visual encoders for downstream tasks, it is often chosen to fine-tune the feature extractor that has been pre-trained with a large-scale dataset using the dataset of the downstream task. Contrastive learning is considered as a very good way of fine-tuning. Training using contrastive learning is usually unsupervised, and the features obtained after training can be efficiently adapted to downstream tasks [9] . Radford et al. [24] proposed the CLIP framework, which is a contrastive learning method for textual and visual modalities. Guzhov et al. [8] adds audio modalities to the CLIP framework to construct the AudioCLIP framework. One of the central determinants of how well contrastive learning works is the definition of positive and negative sample pairs, and our work redefines positive and negative sample pairs to be more consistent with the event localization task. 
method: 
 In this section, we first introduce the problem definition of the audio-visual event localization task. Subsequently, we present our proposed network framework, as depicted in Figure 2 , which consists of three parts: audio-visual co-guidance attention, backgroundevent contrast enhancement, and modal feature fine-tuning.We divide a given video sequence S into 𝑇 non-overlapping segments of one second each, denoted as 𝑠 𝑡 , where S = {𝑠 1 , 𝑠 2 , . . . 𝑠 𝑇 }. Similarly, the corresponding visual and audio sequences can be represented as V = {𝑣 1 , 𝑣 2 , . . . 𝑣 𝑇 } and A = {𝑎 1 , 𝑎 2 , . . . 𝑎 𝑇 }, respectively. The goal of audio-visual event localization is to accurately identify and locate event in those video segments where both visual and audio signals correspond to the same category. It requires the model to not only recognize whether the visual and audio inputs are matched but also accurately predicts the category of event.Specifically, the model needs to predict the category of event associated with the visual segment 𝑣 𝑡 and audio segment 𝑎 𝑡 for each video segment 𝑠 𝑡 within the input video sequence, where 𝑡 ∈ (1,𝑇 ]. Conversely, if the visual and audio does not match, meaning they belong to different categories of event or one of them does not represent any event category, the segment is considered as background. During training, we have access to segment-level labels, denoted as y 𝑡 within 𝑠 𝑡 , which indicate whether the video segment is considered as audio-visual event and which category it belongs to. Concretely,y 𝑡 = 𝑦 𝑐 𝑡 | 𝑦 𝑐 𝑡 ∈ 0, 1, 𝑐 = 1, . . . , 𝐶, 𝐶 𝑐=1 𝑦 𝑐 𝑡 = 1, where 𝐶 is the number of categories. Here, 𝑦 𝑐 𝑡 = 1 indicates the presence of event in 𝑠 𝑡 and its category as 𝑐, while 𝑦 𝑐 𝑡 = 0 indicates a mismatch of audio and visual information, which is regarded as background.Although audio or visual information alone is not always reliable, they can provide each other with instructive information. Therefore, we use audio information to guide the processing of visual signals, which helps extract useful visual information for event localization from complex scene; similarly, we also use visual signals to guide audio processing to reduce the interference of noise mixed in the audio. As shown in Figure 3 , through this audio-visual co-guidance attention (AVCA), the model can more robustly extract audio and visual features directly related to the event. For ease of representation, we denote the visual and audio features of the video segment 𝑠 𝑡 as 𝑣 𝑡 ∈ R 𝑑 𝑣 × (𝐻 * 𝑊 ) and 𝑎 𝑡 ∈ R 𝑑 𝑎 , respectively.Audio-guided spatial-channel visual feature. The work of researchers such as [28, 31] has demonstrated the importance of audio-guided visual features. Following Xu et al. [31] , we obtain audio-guided channel-spatial attentive visual features 𝑣 𝑐𝑠 𝑡 ∈ R 𝑑 𝑣 . However, the reliability of these guided visual features is limited by the quality of the audio signal. Noise in the audio can obstruct the extraction of visual features related to audio-visual event.Consequently, to robustly extract visual regions related to event, inspired by [15, 30] , we utilize global features 𝑣 𝑔 𝑡 ∈ R 𝑑 𝑣 × (𝐻 * 𝑊 ) , obtained from spatial self-attention of the visual signals, to generate channel-level calibration features ṽ𝑔 𝑡 ∈ R 𝑑 𝑣 . These calibration features are used to adaptively modulate the audio-guided visual features, selectively emphasizing useful features while suppressing less useful ones. The calibration features ṽ𝑔 𝑡 can be described as:EQUATIONwhere ⊗ represents matrix multiplication, ⊙ represents elementwise multiplication, W ∈ R Temporal Self-Attention 1 0 EQUATION)where 𝜎 represents the sigmoid function, and 𝛽 is a hyperparameter.Visual-guided enhancement audio feature. Due to inherent noise in audio signals, it is necessary to guide audio features with visual information to reduce audio interference. To extract visual-related features from audio modality, we first employ selfattention on visual information to capture global features v 𝑔 ∈ R 𝑇 ×𝑑 𝑣 × (𝐻 * 𝑊 ) ; unlike spatial-level self-attention for visual modality, we apply temporal self-attention to the audio input, meaning that current time features depend on other moments to derive the global audio features 𝒂 𝑔 , which can be expressed as:𝒂 𝑔 = Softmax 𝑞𝐾 𝑇 𝑉 (3)where 𝑞, 𝐾, 𝑉 can be denoted as:EQUATIONwhere 𝒂 ∈ R 𝑇 ×𝑑 𝑎 represents the audio inputs of video segments. 𝑾 𝑄 , 𝑾 𝐾 , 𝑾 𝑉 ∈ R 𝑑 𝑎 ×𝑑 𝑎 are fully connected layers.With the global spatial visual features v 𝑔 and the temporal global audio features 𝒂 𝑔 , we squeeze the spatial visual features into channel representations, denoted as v 𝑔′ ∈ R 𝑇 ×𝑑 𝑣 . This process effectively encapsulates spatial information within channel dimensions. The squeeze is achieved through the function v 𝑔′ = 𝐹 𝑠𝑞 (v 𝑔 ). Following this, v 𝑔′ is used as a query input to extract audio features related to visual modality, which results in visual-guided audio features ã𝑔 for calibration. It adaptively acquires features from the audio features beneficial for event localization, and suppresses out the features that are not relevant to the event. The audio features ã𝑔 involved in calibration can be calculated according to Equation 3, and corresponding 𝑞, 𝐾, 𝑉 can be expressed as:EQUATION)where𝑾 𝑄 𝑣 ∈ R 𝑑 𝑣 ×𝑑 𝑎 , 𝑾 𝐾 𝑎 , 𝑾 𝑉 𝑎 ∈ R 𝑑 𝑎 ×𝑑 𝑎 denote linear projection for dimensional alignment.The audio output after visual guidance is:EQUATIONwhere 𝜎 represents the sigmoid function, and 𝜓 is a hyperparameter. The ã𝑔 𝑡 denotes the value of ã𝑔 at time 𝑡.After passing through the bi-directional guidance, the audio-visual segments acquire event-related visual representation 𝑣 𝑡 and audio representation 𝑎 𝑡 . To perform effective audio-visual event localization, comprehensive feature fusion of the audio-visual information is necessary. We utilize the method described by Xu et al. [31] , starting with multi-head attention mechanism and residual connections to extract information within each modality. Then, we employ cross-modal relation attention mechanism. Here, features of a single modality are used as the query, while features concatenated across audio-visual dimensions serve as the key and value, which enables cross-modal information fusion without overlooking details within each modality. The feature fusion F 𝑎𝑣 is then obtained through an audio-visual interaction module.To refine the distinction between background and event and acquire features with distinct background-event differentiation, we utilize background-event contrast enhancement (BECE). Concretely, we choose to enhance the fused features using supervised contrastive learning [16] . Specifically, the supervised contrastive learning is conducted within each video sample, where one group of samples for supervised contrastive learning comprises a video along with all its audio-visual segments that have undergone data augmentation. We choose to add Gaussian noise as the method for data augmentation. Each video contains 𝑇 audio-visual pairs, thus a training set consists of 2𝑇 samples. Since the categories of audiovisual event are the same across different segments of the same video, we define positive sample pairs by this way: audio-visual pairs labeled as background and their own augmented ones are positive sample pairs; audio-visual pairs labeled as event and all other audio-visual pairs labeled as event are positive sample pairs.To prevent overfitting, we use only one nonlinear layer to optimize the fused features, serving both as the query and key encoder. The loss function for supervised contrastive learning can be expressed as follows:EQUATIONwhere 𝐼 represents a set of samples for contrastive learning, where 𝑖 ∈ 𝐼 ≡ {1, 2, . . . , 2𝑇 } indexes the samples. 𝐾 (𝑖) denotes the set of positive samples corresponding to the sample indexed by 𝑖, and |𝐾 (𝑖)| is the number of positive samples. 𝑃 (𝑖) ≡ 𝐼 \ 𝑖 represents the set of samples excluding the one indexed by 𝑖. 𝒇 𝑖 represents fused feature corresponding to the sample with index 𝑖, and 𝜏 ∈ R + is the temperature coefficient. The loss function L contrast enhances the optimization within the feature space, promoting the clustering of features labeled as event closer together, while distancing them from the features labeled as background. It encourages more discriminative feature representations for accurate event localization.To preserve the information in the original fused features, we adopt the following method to derive the fused features:EQUATIONwhere F 𝑜 represents the final fused feature used for binary classification between events and background. F 𝑎𝑣 is the original fused feature, and F 𝐹𝑇 is the feature enhanced through supervised contrastive learning. This approach allows for preserving the original feature information while incorporating optimized features obtained through supervised contrastive learning, effectively balancing the model's generalization ability and localization accuracy.We utilize visual and audio encoders, specifically ResNet-50 [10] and Cnn14 [17] , which are pre-trained on large-scale datasets Ima-geNet and AudioSet respectively. They allow for efficient extraction of image and audio features. Given that the Audio-Visual Event (AVE) dataset is a subset of AudioSet [7] , we fine-tune the visual encoder using contrastive learning on this dataset to better adopt to audio-visual event localization tasks. We do average pooling of all the extracted features obtained from one video as samples in the contrastive learning dataset. This process provides each video with a set of visual and audio samples. Visual and audio samples from the same video serve as positive pairs, while samples from different videos serve as negative pairs. The loss function used for contrastive learning is 𝐿 InfoNCE , defined as follows:EQUATIONHere, 𝐵 represents the number of samples in a contrastive learning batch, 𝒇 𝐼 and 𝒇 𝐴 denote visual and audio features respectively, and 𝑖 and 𝑗 are index of visual and audio samples. The temperature parameter 𝜏 ∈ R + influences the separation of the feature space. This loss function ensures that during backpropagation, features of positive pairs become more similar, while features of negative pairs diverge, thereby enhancing the generality of the features.For the final task of audio-visual event localization, we divide it into two subtasks. The first is the model's ability to detect whether the visual and audio information is matched, and the second is the model's ability to accurately determine the category of the event on a video-level basis. For the task of determining whether a video segment is background or event, we use the contrast-enhanced fused feature F 𝑜 ∈ R 𝑇 ×𝑑 . This allows us to obtain the probability scores of event occurrence ŷ = { ŷ1 , ŷ2 , • • • , ŷ𝑇 }:EQUATIONwhere 𝜎 denotes the sigmoid function, and 𝑾 3 ∈ R 1×𝑑 represents a fully connected layer with a single output neuron.For classifying the category of audio-visual event, we use the original fused feature F 𝑎𝑣 ∈ R 𝑇 ×𝑑 . Further, after obtaining videolevel feature through max pooling across the temporal dimension, we connect it to a classification linear layer 𝑾 4 ∈ R 𝑑 ×𝐶 . This setup yields the video-level event category scores ŷ𝑐 as follows:EQUATIONwhere max(F 𝑎𝑣 ) denotes the max pooling operation across the time dimension, extracting the most prominent features for each category, and Softmax is applied to convert the linear layer outputs into probabilities for each event category.During the training phase, since we have access to the event labels y 𝑡 = 𝑦 𝑐 𝑡 | 𝑦 𝑐 𝑡 ∈ 0, 1 , we use y 𝑡 as the label for binary classification between background and event, and y 𝑐 = argmax 𝑐 (y 𝑡 ) as the label for event category classification to train the model.Consequently, the loss function for training the model can be expressed as:EQUATION)where L 𝑐 represents the cross-entropy loss for event category classification, L 𝑒 𝑡 represents the cross-entropy loss for background-event classification, and L 𝑐𝑜𝑛𝑡𝑟𝑎𝑠𝑡 represents the contrastive learning loss. Note that we incorporate the loss L 𝑠𝑢𝑝 𝑡 used for background suppression from [30] to suppress asynchronous audio-visual background within events and enhance audio-visual consistency.During the inference phase, the model's prediction for each video segment 𝑠 𝑡 , denoted as 𝑜 𝑡 , is determined by both ŷ𝑡 and ŷ𝑐 . The prediction is computed as 𝑜 𝑡 = 𝐻 ( ŷ𝑡 -𝜃 ) * ŷ𝑐 , where 𝐻 is the Heaviside step function, and 𝜃 is a threshold for determining the presence of the event. In this work, we set this threshold 𝜃 to 0.5. 
experiments: 
 In this section, we first discuss the experimental setup and then provide a series of ablation studies to show the effectiveness of the various components of the proposed method. Finally, we conduct experiments on the audio-visual event (AVE) dataset to compare our method with current state-of-the-art methods.Audio-visual event dataset. Consistent with existing work [28, [30] [31] [32] , we evaluated our approach on the publicly available AVE dataset. The AVE dataset [28] is a subset of the AudioSet [7] , which contains 4,143 videos with a total of 28 event categories and 1 background category. The AVE dataset covers videos of a variety of realistic scenarios such as musical instruments playing, male speaking and train whistle, etc. Each video has a duration of 10 seconds and contains at least one event category. Each video is divided into 10 segments, each lasting 1 second, with labels assigned to each segment. Each video has at least 2 segments labeled as audiovisual event. In the audio-visual event localization task, the method we propose needs to predict for each video segment whether it is background or event and its event category.Implementation details. For feature extraction, we explored on two types of encoders. One is the configuration in the original work [28] , where visual features are averaged using the "pool5" layer of VGGNet-19 [27] pre-trained on ImageNet [26] for 16-frameper-second in video, and audio features are extracted using the VGGish [11] pre-trained on AudioSet, which converts the audio to the log-spectrogram. The other is the ResNet-50 [10] pre-trained on ImageNet and the Cnn14 network [17] pre-trained on AudioSet. The latter is fine-tuned on the AVE dataset through contrastive learning to obtain more generalized feature representations.In this section, we conduct experiments to validate the effectiveness of each part of the proposed method. In subsequent experiments, we uniformly use visual features extracted by VGG-19 with audio features extracted by VGGish as model inputs.Effectiveness of audio-visual co-guidance attention. In our approach, we implement a bi-directional guidance mechanism based on the fact that visual and audio modalities provide different information for event localization and they access each other's guidance signals to collaborate in cross-modal attentional guidance. Therefore, we compare audio-visual co-guidance attention with three other attentional guidance mechanisms: 1) "w/o AVCA": where visual and audio features are directly fed into the network for integration without any guidance 2)"w/ AVCA-Visual-only": where audio information guide visual modality while preserving the original audio features 3) "w/ AVCA-Audio-only": where visual information guide audio modality while preserving the original visual features. The experimental results are shown in Table 1 , where it can be observed that the performance of the network decreases dramatically when the audio-visual co-guidance attention module is removed, which indicates that AVCA plays an important role in reducing the inconsistency of the inter-modal information and the impact of misinformation. Consistent with existing work, we find that audio-guided visual signals have a nearly 0.6% accuracy improvement for the network compared to no guidance, which suggests that audio signals can be used to allow visual learning to focus on event-related regions.Surprisingly, the method w/ AVCA-Visual-only does not result in performance gain. On the contrary, its inclusion has a negative effect. We attribute the result to the significant background noise present in visual scenes. Directly guiding the audio modality using visual signals may result in the neglect of event-related information in the audio modality, and may even lead to focusing on audio noise instead. Finally, the improvement brought by the audio-visual co-guidance attention is significant. It gives a 2.04% accuracy improvement compared to not using the AVCA method, proving the superiority of the audio-visual co-guidance attention.Effectiveness of background-event contrast enhancement. In our approach, we propose using supervised contrastive learning to optimize the fused feature F 𝑜 for classification between event and background. We first obtain the background-event contrast enhancement fused feature F 𝐹𝑇 and merge it with the original fused feature F 𝑎𝑣 under a certain weight 𝜆. We compare the results under different weights with those obtained without background-event contrast enhancement, as shown in Table 2 . From the table, it can be observed that when the weight is appropriately selected (𝜆 = 0.4 and 𝜆 = 0.6), the model achieves a accuracy improvement of 0.5%, which indicates that supervised contrastive learning effectively help the model distinguish between event and background. However, when the weight is too high (𝜆 = 0.8), the model performance decreases, which suggests that the fine-tuned fused feature does not entirely reflect the visual modality information and also proves the necessity of retaining the original fused feature F 𝑎𝑣 . Furthermore, we test the performance of using the enhanced fused feature under the optimal parameters for both event background binary classification and event classification. The results show that although the model's accuracy increases from 80.30% to 80.65% after enhancement, it still lower than the 80.80% achieved when the enhanced fused features are used exclusively for eventbackground binary classification, indicating that the original features perform better for event classification. We attribute the result to the fact that since the supervisory signal for supervised contrastive learning is the event-background binary classification label, it serves to further distance the event and background features in the feature space, but does not separate the features of different events. Moreover, contrastive learning may even alter the relative positions of features for certain event in the feature space, reducing the distance between them and features of other event categories, thereby leading to a degradation in model performance.Effectiveness of a more efficient encoder. In our approach, we employed ResNet50 and Cnn14 [17] as visual and audio feature extractors, respectively, which have been pre-trained on the ImageNet and AudioSet datasets. These encoders outperform the VGG19 and VGGish models previously used for extracting baseline features, providing more robust feature extraction which contributes to enhanced model performance. Additionally, we fine-tune the visual encoder, ResNet50, to tailor its capabilities more closely to the audiovisual event localization task. We compared these results with those obtained using the original encoder and switching to more efficient encoders, and the outcomes are presented in Table 3 .The results indicate that using more efficient encoders for feature extraction can increase model's accuracy by 0.8%. Furthermore, features extracted using the fine-tuned ResNet50 provided an additional 0.8% accuracy improvement. This not only demonstrates that using more efficient encoders for feature extraction perform better for downstream tasks but also validates the effectiveness of fine-tuning encoders with downstream task datasets. Summary of ablation experiment results. To clearly see the contribution of each part of our proposed method to the model performance, we summarize the experimental results of the three methods, namely, audio-visual co-guidance attention (AVCA), backgroundevent contrast enhancement (BECE), and more efficient encoder, in Table 4 . It can be seen that AVCA, BECE each used individually gives an improvement in model performance. When they are used together, the accuracy of 80.80% is achieved. Compared to the baseline result, i.e., 78.83% accuracy achieved without any of our methods, the accuracy is improved by about 2%, proving the effectiveness of our methods. Further, by adding fine-tuned efficient encoders for feature extraction on AVCA and BECE, our method achieves the best result, i.e., 82.36%.As shown in Table 5 , our proposed CACE-Net model has a significant improvement over existing state-of-the-art methods. Previous methods, such as using only audio or visual unimodal information as model inputs in AVEL, had lower accuracy of 59.5% and 55.3%, respectively. Subsequent models, such as AVEL [28] , DAM [29] , and CMRAN [31] , have integrated audio and visual modalities in more sophisticated manners, resulting in significantly improved accuracy. With the introduction of models such as PSP [32] and CMBS [30] , the accuracy has improved to close to 80%. The CSS [4] model further refines this integration with the accuracy of 80.5%.Our CACE-Net utilizes features extracted from VGG-like and VGG-19 to achieve the accuracy of 80.8%, outperforming all other methods. Additionally, we test the result of CACE-Net when extracting features using more robust encoders, achieving 82.4% accuracy.We show the results of our method on two relatively difficult samples selected from the dataset, as shown in Figure 4 . In the upper samples, the event category is labeled accordion; however, the audio signal is mixed with noise signals that are not relevant to the event, as well as signals that are relevant to other event categories, such as the "female speaking", which interferes with determining the event category. As a result, it is difficult to achieve good results without visual-guided audio features such as CMGA, which fails to predict the accordion event in the last three seconds. In contrast to our method, CACE-Net achieves more accurate prediction results. However, the model still can't predict all correctly, i.e., incorrectly predicting the background as the accordion in 5-7 seconds. We attribute the experimental result to a highly dominant event-related audio signal in 5-7 seconds. Even though overall our method achieves superior discrimination, how this dominant signal can be further identified and optimized deserves further exploration, which we leave to subsequent work. In the sample below, the racing information only appears in the middle 2 seconds of the video, and our model correctly accomplishes this prediction as well.In addition, we visualize different attentional guidance and the results are shown in Figure 5 . The first three figures counting from the left show that our attentional guidance does not attend to nonevent-related information. In the third figure, the event category is the horse, and attention is focused only on the horse itself and not on the man next to it. The last three figures counting from the left show that our attentional guidance is able to more accurately find information related to the event category, which suggests that our audio-visual co-guidance attention reduces inconsistencies in the audio-visual information and attends to the visual regions relevant to event more accurately. 
conclusion: 
 In this paper, we propose an innovative network for audio-visual event localization by exploring multimodal learning to improve the comprehension and prediction of complex audio-visual scenes. Our model reduces the inconsistency of cross-modal information through the audio-visual co-guidance attention mechanism. In addition, we introduce a supervised contrastive learning strategy to improve the distinction between event and background by intentionally perturbing features, which further improves the model's ability to capture fine-grained features. Our experimental results on the task of audio-visual event localization show that CACE-Net can achieve better performance than existing state-of-the-art methods, validating the effectiveness of our proposed strategies in improving the generalization ability and classification accuracy of the models. 
 
 Appendix:  


Decision: The initial ratings are 1 BR and 2 BA, and they are 1 WA and 2 BA after rebuttal. The reviewers all appreciated the proposed method about audio-visual co-guidance attention mechanism, and are satisfied with the authors' responses. 
The response to reviewers should be added in the final version.
I agree with the reviewers and recommend an acceptance.

Reviews: review: paper_summary: paper_summary: The paper presents CACE-Net, a cutting-edge network model designed for audio-visual event localization. The core innovations of CACE-Net include:
1. Audio-Visual Co-Guidance Attention Mechanism: This mechanism facilitates bidirectional guidance between audio and visual inputs, enhancing feature extraction by reducing inconsistencies between the modalities.
2. Background-Event Contrast Enhancement: Incorporates contrastive learning to better distinguish events from the background, leading to more precise and discernible features.
3. Efficient Feature Extractor Fine-Tuning: Utilizes advanced encoders for both audio and visual data, fine-tuned for improved feature extraction from complex multimodal inputs.

Experiments conducted on the AVE dataset demonstrate that CACE-Net outperforms existing state-of-the-art methods, establishing a new benchmark in audio-visual event localization.
strength_weakness: strength_weakness: 1. Novelty: The co-guidance attention mechanism and contrastive enhancement are novel and effectively address key challenges in audio-visual event localization.
2. Technical Correctness: The methods are theoretically sound, with detailed algorithmic descriptions and mathematical formulations supporting their validity.
3. Evaluation: Extensive experiments on the AVE dataset showcase CACE-Net's superiority over current methods. Ablation studies provide clear evidence of the contribution of each component.
4. Clarity: The paper is well-structured and clearly written, making the methodology and results easy to understand.
5. Applications: The model has practical applications in areas such as content analysis, and multimedia retrieval.
limitations: limitations: 1. Generalization: The model's generalizability to other datasets or real-world scenarios with varying types of audio-visual events is not thoroughly explored.
2. Complexity: The bidirectional guidance mechanism and contrastive learning introduce additional complexity, which may pose challenges for real-time applications.
3. Real-world Noises: The paper lacks detailed analysis on how well CACE-Net handles real-world noise and occlusions in visual data.
4. Error Analysis: A detailed error analysis is missing, which could provide insights into the model's failures and areas for improvement.
paper_summary: paper_summary: The paper introduces CACE-Net, a novel approach for audio-visual event localization. This task involves identifying and classifying events in videos by integrating both audio and visual signals. The key contributions include an audio-visual co-guidance attention mechanism and a background-event contrast enhancement strategy. The co-guidance mechanism allows bidirectional attention between audio and visual inputs, improving cross-modal consistency. The contrast enhancement strategy leverages contrastive learning to better distinguish events from backgrounds. The proposed method demonstrates state-of-the-art performance on the AVE dataset, setting a new benchmark for audio-visual event localization.
strength_weakness: strength_weakness: 1. Novelty: The introduction of an audio-visual co-guidance attention mechanism is a significant innovation. Unlike previous methods that primarily use one modality to guide another, this bidirectional approach ensures a more balanced and effective integration of audio and visual information.
2. Technical Approach: The paper presents a well-structured methodology. The co-guidance attention reduces inconsistencies between modalities, while the contrast enhancement fine-tunes the model’s ability to distinguish events from backgrounds. These techniques are backed by rigorous mathematical formulations and justified through empirical results.
3. Adequate Evaluation: Extensive experiments on the AVE dataset validate the effectiveness of CACE-Net. The ablation studies are thorough, demonstrating the contribution of each component to the overall performance.
4. Clarity: The paper is well-written, with clear explanations of complex concepts. The figures and tables effectively illustrate the model architecture and experimental results.
5. Applications: The proposed method has significant implications for multimedia processing, particularly in scenarios requiring robust event localization in videos with varying backgrounds and noise levels.
limitations: limitations: 1. Complexity: While the paper introduces novel mechanisms, the added complexity might be a challenge for practical implementations. The bidirectional attention mechanism and contrastive learning strategy could increase computational overhead.
2. Generalization: Although the method performs well on the AVE dataset, its generalization to other datasets and real-world scenarios is not fully explored. Additional experiments on diverse datasets would strengthen the claim of robustness.
3. Fine-grained Features: The paper mentions the difficulty in distinguishing between similar events and backgrounds. Despite the contrast enhancement strategy, there might still be cases where fine-grained feature extraction falls short, as noted in some qualitative analyses.
4. Comparative Baselines: The comparison with existing methods primarily focuses on performance metrics. A deeper analysis of how CACE-Net handles specific challenging cases compared to baseline methods would provide more insight.
paper_summary: paper_summary: In order to efficiently identify concurrent visual and auditory events from unconstrained videos, locate them, and classify their categories, this paper proposes CACE-Net, an innovative audio-visual event localization network where they design an audio-visual co-guidance attention mechanism and employ a background-event contrast enhancement (BECE) method.
This work has some significance about how to integrate audio and visual modal information and capture fine-grained features for event classification. But more experimental analyses need to be provided, such as considering the impact of using different data augmentation methods on the event-background prediction.
strength_weakness: strength_weakness: 1. A novel audio-visual co-guidance attention mechanism is proposed for effectively reduces the inconsistency of inter-modal information.
2. A background-event contrast enhancement module that uses the contrast learning strategy to randomly perturb the fusion features is proposed for enhancing the contrast between the event segments and the background.
3. More advanced visual and audio encoders are chosen by performing targeted fine-tuning to extract fine features from complex multi-modal inputs.
limitations: limitations: 1. The method of data augmentation is important in BCBE module. Can the authors give the results of using different data augmentation methods on the event-background prediction?
2. In Figure 2, it is better to add an arrow between background feature and the enhanced background feature, which can be consistent with the description of positive pairs in section 3.3.
3. To verify the effectiveness of the targeted fine-tuning encoders, adding a targeted fine-tuning experiments on the original encoders is necessary.
4. In Equation 12, the weights corresponding to different loss functions are the same. Will changing the weights to different values have an impact on the results?
5. The experiments are all conducted on the AVE datasets, and the authors can consider other similar datasets to evaluate the model's generalization ability.