OpenAVE: Moving towards Open Set Audio-Visual Event Localization

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-11 08:07:12

Abstract: Audio-Visual Event (AVE) Localization aims to identify and classify video segments that are both audible and visible, a field that has seen substantial progress in recent years. Existing methods operate under a closed-set assumption and struggle to recognize unknown events in open-world scenarios. To better adapt to real-life applications, we introduce the Open Set Audio-Visual Event Localization task and propose a novel and effective network called OpenAVE based on evidential deep learning. To the best of our knowledge, this is the first effort to address this challenge. Our approach encompasses deep evidential AVE classification and event-relevant prediction, targeting the nuanced demands of open-set environments. Our approach includes deep evidential AVE classification and event-relevant prediction. The deep evidential AVE classification manages event classification uncertainty by extracting class evidence from segment-specific representations enriched with multi-scale context. To effectively distinguish between unknown events and background segments, event-relevant prediction utilizes positive-unlabeled learning. Futhermore, a learnable Gaussian-prior prediction branch is adopted to enhance the performance of event-relevant prediction. Experimental results demonstrate that OpenAVE significantly outperforms state-of-the-art models on the Audio-Visual Event dataset, confirming the effectiveness of our proposed method.

Main paper: 
 introduction: 
 Audio-Visual Event (AVE) Localization task [32] is one of hot topics in the fields of visual-audio scene understanding, which aims at simultaneously determining the presence of an event that is both audible and visible in a video segment from the arbitrarily untrimmed video, and classifying it into a certain event category. It has attracted significant attention from the research community owing to its extensive application potentials, such as video summarization [42] , action recognition [14] , egocentric object detection [19] , indoor navigation [6] and so on. In recent years, a plethora of audio-visual event localization approaches [22, 26, 36, 37, 44] have been proposed and have demonstrated remarkable performance. Despite their success, previous work mainly handle AVE task under the closed set assumption, which classify each event segment into one of the classes encountered during training. This closed set condition limits their application in real-world scenarios, since an input video whose classes are beyond the range of the training set will be misclassified as one of the known categories. Therefore, to tackle this problem, we consider a more challenging and practical AVE setting in this work, termed as Open Set Audio-Visual Event Localization (OSAVE).OSAVE aims to not only recognize known audio-visual events in each video segment but also to reject the unknown ones. As illustrated in Figure 1 , given an untrimmed testing video containing a unknown event (e.g., Baby Cry), traditional AVE methods fails to identify this unknown audio-visual event segment and tends to assign its label to a known class (e.g., Male Speech). Therefore, categories of both known and unknown video segments are expected to be predicted in OSAVE task. Compared with traditional AVE problem, OSAVE is more challenging in two aspects: (1) The temporal nature of videos might result in the diversity and complexity of audio-visual events. Thus, an OSAVE model is required to learn more discriminative event representations of closed set categories but also be aware of what it does not belongs to a known class when the unknown event appears in an open-world scenario. (2) Previous AVE works [37, 38] could rely on segment-level label to correctly predict foreground events and the background. However, due to the lack of unknown event annotations in the open set setting, these existing models fail to effectively differentiate unknown events and pure background from the mixture video segments in the inference stage.In this paper, we propose a novel and effective framework (Ope-nAVE) for open set audio-visual event localization task. To tackle the first challenge, we design a multi-scale context perception module in the deep evidential AVE classification network to obtain more discriminative segment-specific features by taking advantage of richer temporal context, which is not only greatly help for known classification but also provides abundant and distinct information to estimate category uncertainty. To enable the model to know the unknown in the OSVAE task, we formulates it as an uncertainty estimation problem by leveraging evidential deep learning (EDL). EDL could directly use deep neural networks to predict a Dirichlet distribution of event class probabilities, which is informative to quantify the predictive uncertainty of audio-visual events so that the model could discover those high-uncertainty unknown ones.To address the second challenge, we introduce an event-relevant prediction network designed to distinguish between unknown foreground events and background segments. Since the unknown foreground events without annotations are mixed with background segments, learning from labeled known audio-visual events and the mixture can be considered as a semi-supervised OSR problem. Therefore, we apply the positive-unlabeled learning (PU learning) algorithm by training a binary foreground-background classifier to discover potential foreground segments (known and unknown events) in the testing video. Besides, a learnable Gaussian-prior event-relevant branch leveraging local context is proposed to improve the smoothness of learning-based event-relevant scores, thus providing more reliable positive and negative samples for PU learning. Benefiting from evidential deep learning theory and PU learning, our proposed OpenAVE is not only practically flexible to implement but also more effective in distinguishing between known and unknown events. Based on the existing audio-visual event localization dataset (AVE dataset), we construct a new benchmark to evaluate our model and all baselines for OSAVE task. Extensive experiments show that our method outperforms state-of-the-art methods in realistic open-world scenarios. In summary, our main contributions are as follows:• We propose an Open Set Audio-Visual Event localization network to identify unknown events within a video. The event-relevant prediction network mainly applies the positive-unlabeled learning (PU learning) for distinguish unlabeled unknown events from the background. In the inference stage, we leverage the uncertainty and event-relevant scores, video segments from the known and unknown classes, as well as background frames can be effectively distinguished in the OSAVE setting.explored to handle the OSR problem. For example, Neal et al. [24] introduced a counterfactual image generation method that leverages generated data close to training samples but not belonging to known classes to train an open set classifier. Other generative model-based works [25, 30, 39] borrowed the idea of feature reconstruction by utilizing the reconstruction error associated with the generator as an open set indicator to reject unknown ones. Prototype-based learning approaches [7, 9] aim to identify the unknown by calculating the maximum distance between the input example and the learned known class prototypes. Furthermore, probabilistic and evidential deep learning methods [11, 13, 23, 34] [2, 10, 18] and open set object detection/segmentation [12, 20, 21] , yielding promising results. Unlike image samples, audio-visual events entail cross-modal associations and temporal dynamics. Therefore, our proposed Event-relevant prediction mechanism enables the distinction between unknown events and background segments from open-set setting. 
the proposed method 3.1 notations and preliminaries: 
 Problem formulation of OSAVE. The Open Set Audio-Visual Event (OSAVE) localization task aims to predict which temporal segment of an input untrimmed video contains an audio-visual event and to determine the known category to which the event belongs, while also identifying and rejecting segments from novel classes as unknown. Formally, we divide a given video sequencesV into 𝑇 non-overlapping audio and visual pairs V = (𝐴 𝑡 , 𝑉 𝑡 ) 𝑇 𝑡 =1, where each segment is one second long. Here, 𝑉 𝑡 represents the visual content, 𝐴 𝑡 represents its synchronized audio counterpart for the 𝑡-th segment. The ground-truth label for each video segment is denoted as𝑦 𝑐 𝑡 = {𝑦 𝑐 𝑡 |𝑦 𝑐 𝑡 ∈ {0, 1}, 𝐶+1 𝑐=1 𝑦 𝑐 𝑡 = 1} ∈ R 𝐶+1, where 𝑦 𝑐 𝑡 ∈ {0, 1} indicates whether an event of category 𝑐 is present in the 𝑡-th segment. 𝐶 is the total number of known event categories, and 𝐶 + 1 represents the background. 𝑦 𝑐 𝑡 = 1 indicates the presence of an event of category 𝑐 in the 𝑡-th segment, while 𝑦 𝑐 𝑡 = 0 represents its absence. The label for the entire video can be denoted as𝑌 = {𝑦 𝑐 1 , 𝑦 𝑐 2 , • • • , 𝑦 𝑐 𝑇 } ∈ R 𝑇 ×𝐶+1 .It is worthy noting that the model only has access to the video data and the annotations of known events during training while the annotations of unknown events are not provided. During the inference phase, the learned model is required to predict a set of the event labels { ŷ c 𝑡 } 𝑁 𝑡 =1 , where 𝑁 is the number of the input video segments in V. Here, c represents an event category, where c ∈ {1, 2, • • • , 𝐶 + 1, 𝑈 }, and 𝑈 denotes the unknown event class. In summary, this task involves labeling video data with audiovisual events, categorizing known events, and distinguishing them from unknown events based on the trained model's predictions. Audio and visual representations. Following previous works [32, 37, 38] , we also utilize pre-trained CNN models to extract audiovisual features {𝑓 𝑎 𝑡 , 𝑓 𝑣 𝑡 } for each segment (𝐴 𝑡 , 𝑉 𝑡 ) in the video V, where 𝑡 ∈ {1, 2, 3, • • • ,𝑇 }. For the audio sequence input, we exploit the VGG-like network pre-trained on AudioSet to extract audio features {𝑓 𝑎 𝑡 } 𝑇 𝑡 =1 ∈ R 𝑇 ×𝑑 𝑎 , where 𝑑 𝑎 denotes the dimension of audio feature vectors. Similarly, the visual representations [32] to find visual regions that are relevant to the audio signals in each video segment. Then, the visual and audio features after AGVA are further input to two convolutional layers with ReLu to project the features from two modalities into the same embedding space. The processed audio and visual features are represented as{𝑓 𝑣 𝑡 } 𝑇 𝑡 =1 ∈ R 𝑇 ×𝑑 𝑣 × ({ f 𝑎 𝑡 } 𝑇 𝑡 =1 ∈ R 𝑇 ×𝐷 and { f 𝑣 𝑡 } 𝑇 𝑡 =1 ∈ R 𝑇 ×𝐷, respectively. Finally, the average features of f 𝑎 𝑡 and f 𝑣 𝑡 are used as the joint multi-modality representation 𝑓 𝑎𝑣 𝑡 ∈ R 1×𝐷 for labeling the video segment.The architecture of our approach primarily consists of two main components: a deep evidential AVE classification network and an event-relevant prediction network. Given an untrimmed video, the deep evidential AVE classification network is tasked with identifying and classifying audio-visual events by leveraging evidential deep learning to manage uncertainty and enhance reliability. This network processes both audio and visual streams to extract features that are combined to form a comprehensive multi-modal understanding of each event segment. Concurrently, the eventrelevant prediction network focuses on determining the relevance of each video segment to the detected events. This network employs positive-unlabeled learning to distinguish between segments that contain relevant events and those that do not, effectively filtering out irrelevant or background segments. By integrating these two networks, our approach not only accurately classifies known audiovisual events but also adeptly identifies segments containing new or previously unrecognized event types, enhancing the model's applicability in dynamic, real-world environments where unseen events may occur. As depicted in Figure 2 , we first extract audio and visual features{ f 𝑎 𝑡 } 𝑇 𝑡 =1 and { f 𝑣 𝑡 } 𝑇 𝑡 =1, respectively, and then integrate them to generate multi-modality presentations {𝑓 𝑎𝑣 } 𝑇 𝑡 =1 . Next, the multi-modality features {𝑓 𝑎𝑣 𝑡 } 𝑇 𝑡 =1 are sent to deep evidential AVE classification network and event-relevant prediction network respectively to jointly determine known and unknown events in the inference stage. In the deep evidential AVE classification network, we first apply a multi-scale context perception module to enable individual segment to integrate rich temporal contextual information from neighborhood video segments. Then the evidence collector 𝑓 on top of the multi-scale context perception module is used to predict class-wise evidence, which formulates a Dirichlet distribution so that event classification probabilities 𝑝 and the predictive uncertainty 𝑢 of each video segment can be determined. To distinguish unknown foreground events from the background, the event-relevant prediction network is learned by the positive-unlabeled learning (PU learning). To further enhance the event-relevant scores S𝑟 𝑡 (predicting the foreground event and background) to construct positive and negative sets in PU learning, we add the Gaussian-prior eventrelevant prediction branch to improve the smoothness of predicted video segments. Moreover, a consistency loss is utilized to reduce the disagreement between the learning-based event-relevant scores 𝑠 𝑟 𝑡 and Gaussian-prior event-relevant scores 𝑔 𝑟 𝑡 . We now detail the specifics of our model. Although some previous AVE approaches [15, 29, 32, 37, 44] adopt Bi-LSTMs [41] or Transformer [33] to establish temporal relationship in the uni-modal or cross-modal for localizing short as well as long-scale events, these model severely ignore the importance of local contextual perception. In fact, neighboring video segments of an event commonly contain more critical contextual cues related to it compared to distant video segments. Moreover, due to an audio-visual event may range diverse duration, sensing temporal contexts in diverse ranges is very essential. Therefore, we argue that endowing individual video segment with the ability of perceiving multi-scale local temporal contexts from neighborhood video segment is very crucial for AVE task.To this end, a multi-scale context perception module (MSCPM) is proposed to enhance each individual event representation by making use of multi-scale temporal contextual cues, which is illustrated in Figure 3 . Specifically, we assign each video segment (𝐴 𝑡 , 𝑉 𝑡 ) with a series of 𝑡-centered multi-scale video segments that provide different temporal context-aware features to extend segment-specific perception ranges. Each 𝑡-centered multi-scale audio-visual segments are denoted as temporal sequence set 𝑍 𝑡 = {(𝑠 𝑚 𝑡 , 𝑒 𝑚 𝑡 )|𝑚 ∈ [1, 𝑀])}, where 𝑀 is the maximum of temporal extension range, 𝑠 𝑚 𝑡 = max(0, 𝑡 -𝑚) and 𝑒 𝑚 𝑡 = min(𝑡 + 𝑚,𝑇 ) represents the starting and ending points of the temporal boundary, respectively. After, we extract the average features 𝑓 𝑚 𝑡 of every temporal sequence (𝑠 𝑚 𝑡 , 𝑒 𝑚 𝑡 ) ∈ 𝑍 𝑡 , which is defined as:EQUATIONTo extend the perception range of the audio-visual segment (𝐴 𝑡 , 𝑉 𝑡 ), we apply feature enhancement to integrate its audio-visual representation and each temporal context 𝑓 𝑚 𝑡 . In general, each individual event gains useful contextual information mainly from extended video segments that are highly relevant to it. To effectively implement feature enhancement and reduce the interference of irrelevant context, this study proposes a weighted feature fusion mechanism. To be specific, we first measure the similarity between the segment features 𝑓 𝑎𝑣 𝑡 and each extended video sequence features {𝑓 𝑚 𝑡 } 𝑀 𝑚=1 , and achieve the cosine similarities 𝜔 𝑚 𝑡 , which can be given by:EQUATIONThen the 𝑓 𝑎𝑣 𝑡 and average feature representations of each weighted extended video segments are fused, and we obtain new audio-visual features 𝑓 𝑎𝑣 𝑡 with extended-range perception, which is formulated as:EQUATIONWhere 𝜓 1 and 𝜓 2 are two fully-connected layers for feature embedding, respectively; 𝛿 represents the Softmax operation with a temperature factor. 𝛾 𝑡 ∈ [0, 1] is the fusion weight representing the scaled average of the cosine similarities {𝜔 𝑚 𝑡 } 𝑀 𝑚=1 , which can be calculated by: EQUATIONEQUATIONWhere 𝑦𝑐 is an one-hot 𝐶-dimensional label for a video segment (𝐴 𝑡 , 𝑉 𝑡 ) and 𝑆 (𝑡 ) = 𝐶 𝑗 𝛼 𝑗 is the total strength over 𝐶 event categories. Based on Subjective Logic theory (SL) and the evidence theory, 𝛼 𝑗 is linked to the learned evidence 𝑒 𝑗 by the equality 𝛼 𝑗 = 𝑒 𝑗 +1, where 𝑒 𝑗 ∈ R 𝐶 + can be represented as 𝑒 𝑗 = 𝑔(𝑓 ( 𝑓 𝑎𝑣 𝑡 ; 𝜃 )). Here, 𝑓 can be seen as an evidence collector, which represents the output of a deep neural network (DNN) parameterized by 𝜃 . 𝑔 is the evidence function, e.g., Exp, Softplus or ReLU, to keep the collected evidence 𝑒 𝑗 non-negative. In the testing, the expected classification probability of each event category is E[𝑝 𝑗 ] = 𝛼 𝑗 /𝑆 and the classification uncertainty 𝑢 is estimated by 𝑢 𝑗 = 𝐶/𝑆. E[•] is to take the mean loss values over the input samples. Note that the uncertainty 𝑢 is inversely proportional to the total evidence of all known categories, therefore it reflects the probability that a video segment belongs to the unknown event.When a given video contains unknown events, the mixture of unknown foreground segments and pure background makes it challenging for the model to distinguish between them solely through uncertainty-aware classification. Therefore, predicting event-relevant scores that indicates the likelihood of a video segment is the foreground is crucial. We also notice the fact that samples from known categories are positive while the mixture of 'background' includes 'positive-unlabeled' data. This intrinsically is a semi-supervised learning problem referred to as positive and unlabelled learning (PU learning) which relies on learned knowledge from positive data to relabel unknown samples. To accurately identify the positive unknown samples from the 'background', a simple but very effective PU learning method is leveraged in this paper.We utilize 𝑠 𝑟 𝑡 ∈ [0, 1] generating from the learning-based eventrelevant prediction to represent the predicted event-relevant score of each audio-visual pair (𝐴 𝑡 , 𝑉 𝑡 ). Note that the learning-based event-relevant prediction branch utilizes a simple binary classifier, consisting of a 2-dimensional fully-connected layer with the sigmoid function, to differentiate between foreground events and the background. A binary cross-entropy (BCE) loss is exploited as the training loss of event-relevant prediction network, which can be denoted as:EQUATIONwhere P and N are the positive and negative sets in a training batch. The positive set P = {𝑠 𝑟 𝑡 |𝑦 𝑐 𝑡 ≤ 𝐶} directly consists of the data belong to known classes while the negative samples are difficult to determine owing to unknown events appearing in unlabeled data. To tackle this problem, we denote those samples {𝑠 𝑟 𝑡 |𝑦 𝑐 𝑡 = 𝐶 + 1} as the unlabeled background set U, and then sort the U in ascending order. Finally, the top-𝑘 examples from the U are selected to construct the most likely negative set N . This simple loss of event-relevant prediction network will push the probably pure background segments far away from positive audio-visual event pairs. Since the learning-based event-relevant score 𝑠 𝑟 𝑡 is typically determined on a segment-level basis without leveraging contextual information from neighboring video segments, we are considering utilizing local context to enhance it. This approach aims to achieve more reliable event-relevant prediction and further improve foreground discrimination. The primary motivation behind using local context is that the predicted event-relevant results should keep a locally consistency. To implement this prior, we introduce the learnable Gaussian masks to generate event-relevant scores. We mainly add a additional Gaussian-prior event-relevant prediction branch in the event-relevant prediction network, which predicts Gaussian kernels (𝜎 𝑡 , 𝜇 𝑡 ) 𝑇 𝑡 =1 to model event-relevant scores for each segment in the video. We obtain a set of segment-specific local Gaussian masks 𝐺 = {𝐺 𝑗 } 𝑇 𝑡 =1 by the predicted Gaussian parameters (𝜎 𝑡 , 𝜇 𝑡 ) 𝑇 𝑡 =1 . The Gaussian-prior event-relevant score 𝑔 𝑟 𝑡 is generated by choosing the values from the 𝑡-th Gaussian mask 𝐺 𝑡 , which is formalized as:EQUATION)Where 𝛽 represents the variance of the segment-specific local Gaussian masks 𝐺. Although Gaussian-prior event-relevant scores {𝑔 𝑟 𝑡 } 𝑇 𝑡 =1 can provide locally smooth results for predicted audiovisual pairs, directly integrating them with the learning-based eventrelevant scores {𝑠 𝑟 𝑡 } 𝑇 𝑡 =1 will make the model worse. We assume that this issue occurs due to the disagreement between these two eventrelevant scores on the same time step. To address this problem, an event-relevant consistency loss L 𝑐𝑜𝑛 is proposed, which is defined as:L 𝑐𝑜𝑛 = ∑︁ 𝑡 (𝑠 𝑟 𝑡 -𝑔 𝑟 𝑡 ) 2 (8)Under the help of the consistency loss, the learnable Gaussianprior event-relevant scores could be effectively complemented to learning-based event-relevant scores to improve the smoothness of predicted segments and provide more accurate foreground predictions. The average scores of 𝑠 𝑟 𝑡 and 𝑔 𝑟 𝑡 are fused to produce s𝑟 𝑡 , which replaces original learning-based event-relevant score 𝑠 𝑟 𝑡 as the new event-relevant score. This new event-relevant score is used to construct positive and negative sets for PU learning. Finally, we optimize the Event-relevant Prediction Network, as shown in Figure 2 , using the combined loss L 𝐸𝑅 = L 𝑐𝑜𝑛 + L 𝑟 .training. By combining all the optimization objectives defined by Eqs.(5)(6)(8), the final weighted sum of multi-task training loss L 𝑡𝑜𝑡𝑎𝑙 is obtained as:EQUATION)Where 𝜆 is a hyperparameter to balance training loss.Inference. For a given untrimmed video, each audio-visual segment input is successively fed into the trained OpenAVE model, which generates the classification labels ŷ𝑡= arg max 𝑗 ∈ [1,2,••• ,𝐶 ] E[𝑝 𝑡 𝑗 ],the classification uncertainty 𝑢 𝑡 and an event-relevant prediction score s𝑟 𝑡 . Relying on the obtained 𝑢 𝑡 and s𝑟 𝑡 , a positive foreground audio-visual segment (s 𝑟 𝑡 > 0.5) can be accepted as the known class label ŷ𝑡 if the 𝑢 𝑡 < 𝜏, else it is rejected as the unknown, where 𝜏 represents a pre-defined outlier threshold. The entire inference procedure of our model is shown in Algorithm 1. 
experiments 4.1 experimental setup: 
 Datasets. We evaluate our method and all baseline models on a public audio-visual event localization benchmark dataset: the Audio-Visual Event (AVE) dataset. The AVE dataset is a subset of the AudioSet [16] , which contains 4,143 videos covering 28 event categories, e.g., airplane flying, dog barking and church bell. Each video lasts for 10 seconds and contains at least one 2-second audiovisual event. The dataset is divided into 3,339 training videos, 402 validation videos, and 402 testing videos. To facilitate open set evaluation, we randomly select 3/4 of the event categories from the AVE training set as known data, while retaining all AVE validation and testing data containing both known and unknown categories. Prediction Set Ŷ = { ŷ c 1 , ŷ c 2 , ŷ c 𝑡 , • • • , ŷ c𝑁 } in the video V. 1: Video data pre-processing. 2: Predict the closed set prediction score 𝑝 𝑡 𝑗 , event-relevant scores s𝑟 𝑡 and classification uncertainty 𝑢 𝑡 of each audio-visual pair(𝐴 𝑡 , 𝑉 𝑡 ) 𝑁 𝑡 =1 in the video V. 3: while {(𝐴 𝑡 , 𝑉 𝑡 )} 𝑁 𝑡 =1 ∈ V do 4:if s𝑟 𝑡 < 0.5 then the video segment (𝐴 𝑡 , 𝑉 𝑡 ) is a background and ŷ c 𝑡 = 𝐶+1; continue.end if end if 9: end while This random selection process is repeated three times, resulting in three different open set data splits. Detailed dataset information is provided in our supplementary materials. Evaluation metrics. In the experiment, the evaluation metrics are divided into closed set and open set metrics. For closed set evaluation metric, we follow previous works [32, 37] and utilize the overall classification accuracy (ACC) to assess closed set AVE performance. To adapt OSAVE evaluation, we introduce the Area Under the Receiver Operating Characteristic (AUROC) curve and the Area Under the Precision-Recall (AUPR) as open set evaluation metrics. These metrics evaluate the performance of detecting unknown events from known events. Additionally, we apply the False Alarm Rate at True Positive Rate of 95% (FAR@95) to address practical operational implications. Implementation details. We use the ResNet-151 [17] and VGG-19 [28] pre-trained on ImageNet as the visual feature extractor, respectively. For each one-second video segment, We sample 16 frames and extract their visual features respectively. Then, we use the average feature maps of these frames to produce segment-level visual feature maps, resulting in the visual dimension of 7 × 7 × 2048 or 7 × 7 × 512 for each segment. For audio representations, we first process the raw audio into log-mel spectrograms. We separately train our method and all baselines on three different splits of the AVE training set and evaluate their performance on the testing dataset.6: if 𝑢 𝑡 < 𝜏 then the video segment (𝐴 𝑡 , 𝑉 𝑡 ) is a known class by ŷ c 𝑡 = arg max 𝑗 E(𝑝 𝑡 𝑗 ), 𝑗 ∈ [1, • • • , 𝐶].The experimental results on the AVE dataset are reported in Table 1 . These results clearly demonstrate that our approach, OpenAVE, consistently and significantly outperforms all comparative models by large margins across both closed and open set AVE metrics. For instance, using the VGG-19 backbone, our method achieves the best AUROC score of 85.62% on AVE dataset split III, surpassing the state-of-the-art ARPL score of 72.02%. Although OpenMax and RPL are also considered recent state-of-the-art approaches for Open Set Recognition (OSR), it is evident that their performances lag considerably behind our method. Notably, the closed-set accuracy (ACC) of OpenMax is dramatically lower compared to other models. This is attributed to OpenMax's strategy of modifying the activation function before the softmax layer and introducing a novel unknown class, which could adversely affect the predictions of known events. When utilizing ResNet-151, our model shows even higher gains, improving AUROC scores by 7.24% on AVE dataset split II compared to ARPL. Despite variations in experimental outcomes across different splits of the AVE dataset, these comparisons robustly highlight the superior performance of OpenAVE in tackling the Open Set Audio-Visual Event Localization (OSAVE) task. 4 . The extension ranges, denoted by 𝑀, were varied from one to nine based on the number of video segments. Our model achieved optimal performance across all metrics when 𝑀 was set to 3. However, as the extension ranges increased, the model's performance on metrics began to decline. This decline can be attributed to the aggregation of more event-irrelevant contextual information into the current video segment as the temporal extension scale increases, thereby impeding the model's ability to extract valuable classification evidence to quantify event predictive uncertainty. 4 , we can see that the performance declines by 1.80% on the AUROC metric without the consistency loss. This further verifies our hypothesis that the two different kinds of event relevant prediction scores require to be aligned for the best performance.  
conclusion: 
 In this paper, we investigate the Open Set Audio-Visual Event Localization (OSAVE) task for the first time, which involves recognizing known events while simultaneously rejecting unknown audiovisual events in open-world scenarios. The OSAVE task presents greater challenges than the traditional AVE task due to the presence of both background segments and unknown foreground events, compounded by the inherent uncertainty of the events. To address this, we propose a novel network for open set audio-visual event localization, which comprises an event-relevant prediction network and a deep evidential AVE classification network. In the proposed OpenAVE, the deep evidential AVE classification network utilizes Evidential Deep Learning (EDL) to manage event classification uncertainty and employs a Multi-Scale Context Perception Module (MSCPM) to derive segment-specific representations that provide more discriminative evidence for the EDL classifier. The eventrelevant prediction network learns to distinguish unknown events from the background using the Positive-Unlabeled learning algorithm (PU learning). Futhermore, a learnable Gaussian-prior branch is integrated into the event-relevant prediction network to enhance the reliability of positive and negative samples for PU learning.Extensive experiments on the Audio-Visual Event benchmark (AVE dataset) demonstrate that our approach achieves state-of-the-art performance in OSAVE. 
 
 Appendix:  


Decision: All the reviewers are satisfied with the response. The final paper should incorporate the author's valid responses. I am delighted to recommend the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: Existing works on audio-visual event localization task are operated under a closed-set assumption and struggle to recognize unknown events in open-world scenarios. To this end, this paper studies the Open Set Audio-Visual Event localization (OSAVE) task and proposes the OpenAVE framework which is based on evidential deep learning. Experimental results demonstrate the effectiveness of the proposed task.
strength_weakness: strength_weakness: - The studied open set audio-visual event localization task is novel and valuable. 
- In addition to the new task, this paper also contributes a strong baseline method that achieves satisfactory open-set event localization.
limitations: limitations: - Some contents in the paper should be more rigorous. For example, the videos in the AVE dataset are 10 seconds, which may not be 'untrimmed' as the authors claimed in Line 48. In Lines 102-104, the authors wrote that 'given an untrimmed testing video containing a unknown event (e.g., Baby Cry), traditional AVE methods fails to identify this unknown audio-visual event segment and tends to assign its label to a known class (e.g., Male Speech).' It would be better if there are some detailed experimental evidences. For example, the authors may select two or three traditional models and report their FAR@95 under the claimed scenarios. 
- It would be better to include all the other evaluation metrics in Table 4. Besides, some references related to the datasets for feature extraction, baselines for comparison, and evaluation metrics should be cited appropriately.
- In Table 3, the authors compare the proposed MSCPM module with the Bi-LSTM and Transformer. In my opinion, the proposed method can also be adapted to be plug-and-play by replacing the MSCPM using existing AVE backbones. If so, it would be better to supplement more comparison results to enhance this part.
- The evidential deep learning used in the proposed method seems to be widely adopted in those temporal action localization [2, 10, 11, 18] methods. The key differences or insights should be emphasized more to demonstrate the novelty of the proposed method better. 
- This paper currently contains many grammatical errors. I only list a few of them here: Line 421- 'these model', Line 555 - 'add a additional',  Line 6 - 'TxC+1', last sentence in the caption of Fig. 2.
- Since this paper involves three dataset splits, the source codes are essential for readers.
paper_summary: paper_summary: The paper addresses the open-domain AVE (Audio-Visual Event) problem, which involves training a model using a labeled training set and then recognizing events in video data that have a different distribution from the training set. This problem avoids excessive reliance on the training data, offering broader practical applicability.
strength_weakness: strength_weakness: The paper achieves the final discrimination of video segments by introducing both event prediction and clustering simultaneously. By aggregating the results of the discriminative outcomes, the model avoids relying solely on the prediction module and excessively on true labels, which leads to poor model generalization.
limitations: limitations: In the classification module, the authors incorporate a multi-scale event perception module to maintain event continuity by jointly leveraging contextual features. However, applying multi-scale processing to all segments to be recognized reduces model efficiency. It might be worth considering increasing multi-scale computations only for uncertain segments based on the prediction module's outcomes.
The authors obtain new unknown class discrimination results by integrating features, which may lead to the erroneous recognition of complex backgrounds as events. It could be beneficial to perform discrimination across different modalities before fusion to gain complementary multi-modal information.
paper_summary: paper_summary: The proposed method significantly enhances performance in the field of Open Set Audio-Visual Event Localization (OSAVE) by effectively recognizing known events while rejecting unknown ones in open-world scenarios. The introduced model, OpenAVE, integrates an event-relevant prediction network and a deep evidential AVE classification network, effectively managing event classification uncertainty and distinguishing between known and unknown events.
strength_weakness: strength_weakness: This paper is well written.
limitations: limitations: In the “RELATED WORK” section, while the paper offers a comprehensive overview of existing methods in Open Set Recognition (OSR), it would benefit from providing more direct comparisons between different approaches, particularly in terms of their effectiveness in handling uncertainty and unknown classes. This would help readers better understand the strengths and limitations of each method and provide clearer justification for the proposed Event-relevant prediction mechanism.

In the "The Proposed Method" section, it would be beneficial to provide a more detailed explanation of why the Dirichlet distribution is employed and how it enhances the method's effectiveness. By elaborating on specific instances or scenarios where the Dirichlet distribution effectively models uncertainty in audio-visual event localization, readers can better understand its practical advantages and the rationale behind its utilization in the proposed approach.

In the “The Proposed Method” section, the inclusion of Gaussian-prior event-relevant scores enhances local context awareness. However, authors could further elucidate how these scores complement the learning-based event-relevant scores to improve overall predictions, providing clarity on the synergy between the two approaches.
In the "EXPERIMENTS" section, the study evaluates the effectiveness of components in the OpenAVE model. Results highlight MSCPM's key role, yielding a substantial AUROC improvement. ERP and GP also contribute significantly. MSCPM outperforms Bi-LSTM and Transformer, leveraging multi-scale temporal contexts. However, further clarification is needed on the impact of different temporal context scales.

In the "EXPERIMENTS" section, the analysis on the consistency loss demonstrates its importance, as the AUROC metric drops by 1.80% without it. This supports the need for aligning different event-relevant prediction scores. However, more detailed reasoning on why this alignment improves performance would be beneficial.

In the "CONCLUSION" section, the summary effectively highlights the contributions and results of the study. However, it would be beneficial to include specific numerical results to quantify the improvement and reinforce the claims of state-of-the-art performance. Additionally, discussing potential future work or limitations could provide a more comprehensive conclusion.
paper_summary: paper_summary: The paper introduces "OpenAVE," a novel framework for Open Set Audio-Visual Event (OSAVE) Localization, addressing the challenge of recognizing and classifying video segments containing concurrent audible and visual cues in real-world, open-world scenarios. This is a pioneering work in the field, as it shifts the focus from traditional closed-set assumptions to handling unknown events. The OpenAVE model incorporates two key components: a deep evidential AVE classification network and an event-relevant prediction network. The former utilizes evidential deep learning combined with multi-scale context to manage classification uncertainty for known events while also identifying unknown ones. The latter employs positive-unlabeled (PU) learning to distinguish unknown foreground events from background segments, further enhanced by a learnable Gaussian-prior branch for smoother learning-based event relevance scores. Comprehensive evaluations on the Audio-Visual Event (AVE) dataset demonstrate that OpenAVE surpasses state-of-the-art models, validating its efficacy for OSAVE.
strength_weakness: strength_weakness: Innovation: The paper presents the first dedicated solution for open set audio-visual event localization, filling a crucial gap in the literature where most methods operate under closed-set constraints.
Evidential Deep Learning Integration: By leveraging evidential theory, OpenAVE provides a robust way to quantify uncertainty in event classification, which is particularly valuable in the presence of unseen events.
Event-Relevant Prediction Network: The introduction of PU learning, alongside the Gaussian-prior branch, enhances the model's ability to discern unknown events from backgrounds, contributing to more accurate open-world event detection.
Comprehensive Evaluation: The authors establish a new benchmark using the AVE dataset for the OSAVE task, showcasing the superiority of OpenAVE through extensive experiments under realistic conditions.
limitations: limitations: There is repeated content in the abstract. “Our approach encompasses…prediction.” V.S. “Our approach includes…prediction.”
Generalizability Concerns: While the proposed model shows strong performance on the AVE dataset, the generalizability to other datasets or real-life scenarios with different event distributions and noise levels is not thoroughly explored.
Complexity and Scalability: The deep evidential and event-relevant prediction networks, while effective, may introduce computational complexity that could limit scalability for large-scale or real-time applications.
Unknown Event Understanding: Although the model distinguishes unknown events from known ones and background, it does not inherently provide insights into what these unknown events might be or how they can be characterized, limiting its utility for event discovery and taxonomy expansion.
Lack of Qualitative Analysis: While quantitative results are comprehensive, qualitative examples demonstrating how the model performs in challenging cases or the types of errors it makes would enrich the understanding of its capabilities.
Overall, OpenAVE represents a significant step forward in audio-visual event localization by tackling open-set recognition, offering a solid foundation for future advancements in understanding complex multimedia content in open-world settings. However, further research could address its limitations, focusing on enhancing scalability, exploring event characterization for unknowns, and validating performance across diverse datasets and real-life scenarios.