Cross-Attentional Audio-Visual Fusion for Weakly-Supervised Action Localization

1 INTRODUCTION . The goal of this paper is to temporally localize actions and events of interest in videos with weaksupervision . In the weakly-supervised setting , only video-level labels are available during the training phase to avoid expensive and time-consuming frame-level annotation . This task is of great importance for video analytics and understanding . Several weakly-supervised methods have been developed for it ( Nguyen et al. , 2018 ; Paul et al. , 2018 ; Narayan et al. , 2019 ; Shi et al. , 2020 ; Jain et al. , 2020 ) and considerable progress has been made . However , only visual information is exploited for this task and audio modality has been mostly overlooked . Both , audio and visual data often depict actions from different viewpoints ( Guo et al. , 2019 ) . Therefore , we propose to explore the joint audio-visual representation to improve the temporal action localization in videos . A few existing works ( Tian et al. , 2018 ; Lin et al. , 2019 ; Xuan et al. , 2020 ) have attempted to fuse audio and visual modalities to localize audio-visual events . These methods have shown promising results , however , these audio-visual events are essentially actions that have strong audio cues , such as playing guitar , and dog barking . Whereas , we aim to localize wider range of actions related to sports , exercises , eating etc . Such actions can also have weak audio aspect and/or can be devoid of informative audio ( e.g . with unrelated background music ) . Therefore , it is a key challenge to fuse audio and visual data in a way that leverages the mutually complementary nature while maintaining the modality-specific information . In order to address this challenge , we propose a novel multi-stage cross-attention mechanism . It progressively learns features from each modality over multiple stages . The inter-modal interaction is allowed at each stage only through cross-attention , and only at the last stage are the visuallyaware audio features and audio-aware visual features concatenated . Thus , an audio-visual feature representation is obtained for each snippet in videos . Separating background from actions/events is a common problem in temporal localization . To this end , we also propose : ( a ) foreground reliability estimation and classification via open-max classifier and ( b ) temporal continuity losses . First , for each video snippet , an open-max classifier predicts ∗Qualcomm AI Research is an initiative of Qualcomm Technologies , Inc. scores for action and background classes , which is composed of two parallel branches for action classification and foreground reliability estimation . Second , for precise action localization with weak supervision , we design temporal consistency losses to enforce temporal continuity of actionclass prediction and foreground reliability . We demonstrate the effectiveness of the proposed method for weakly-supervised localization of both audio-visual events and actions . Extensive experiments are conducted on two video datasets for localizing audio-visual events ( AVE1 ) and actions ( ActivityNet1.22 ) . To the best of our knowledge , it is the first attempt to exploit audio-visual fusion for temporal localization of unconstrained actions in long videos . 2 RELATED WORK . Our work relates to the tasks of localizing of actions and events in videos , as well as to the regime of multi-model representation learning . Weakly-supervised action localization : Wang et al . ( 2017 ) and Nguyen et al . ( 2018 ) employed multiple instance learning ( Dietterich et al. , 1997 ) along with attention mechanism to localize actions in videos . Paul et al . ( 2018 ) introduced a co-activity similarity loss that looks for similar temporal regions in a pair of videos containing a common action class . Narayan et al . ( 2019 ) proposed center loss for the discriminability of action categories at the global-level and counting loss for separability of instances at the local-level . To alleviate the confusion due to background ( nonaction ) segments , Nguyen et al . ( 2019 ) developed the top-down class-guided attention to model background , and ( Yu et al. , 2019 ) exploited temporal relations among video segments . Jain et al . ( 2020 ) segmented a video into interpretable fragments , called ActionBytes , and used them effectively for action proposals . To distinguish action and context ( near-action ) snippets , Shi et al . ( 2020 ) designed the class-agnostic frame-wise probability conditioned on the attention using conditional variational auto-encoder . Luo et al . ( 2020 ) proposed an expectation-maximization multi-instance learning framework where the key instance is modeled as a hidden variable . All these works have explored various ways to temporally differentiate action instances from the near-action background by exploiting only visual modality , whereas we additionally utilize audio modality for the same objective . Audio-visual event localization : The task of audio-visual event localization , as defined in the literature , is to classify each time-step into one of the event classes or background . This is different from action localization , where the goal is to determine the start and the end of each instance of the given action class . In ( Tian et al. , 2018 ) , a network with audio-guided attention was proposed , which showed prototypical results for audio-visual event localization , and cross-modality synchronized event localization . To utilize both global and local cues in event localization , Lin et al . ( 2019 ) conducted audio-visual fusion in both of video-level and snippet-level using multiple LSTMs . Assuming single event videos , Wu et al . ( 2019 ) detected the event-related snippet by matching the video-level feature of one modality with the snippet-level feature sequence of the other modality . Contrastingly , our cross-attention is over the temporal sequences from both the modalities and does not assume single-action videos . In order to address the temporal inconsistency between audio and visual modalities , Xuan et al . ( 2020 ) devised the modality sentinel , which filters out the eventunrelated modalities . Encouraging results have been reported , however , the localization capability of these methods has been shown only for the short fixed-length videos with distinct audio cues . Differently , we aim to fuse audio and visual modalities in order to also localize actions in long , untrimmed and unconstrained videos . Deep multi-modal representation learning : Multi-modal representation learning methods aim to obtain powerful representation ability from multiple modalities ( Guo et al. , 2019 ) . With the advancement of deep-learning , many deep multi-modal representation learning approaches have been developed . Several methods fused features from different modalities in a joint subspace by outerproduct ( Zadeh et al. , 2017 ) , bilinear pooling ( Fukui et al. , 2016 ) , and statistical regularization ( Aytar et al. , 2017 ) . The encoder-decoder framework has also been exploited for multi-modal learning for image-to-image translation ( Huang et al. , 2018 ) and to produce musical translations ( Mor et al. , 1https : //github.com/YapengTian/AVE-ECCV18 2http : //activity-net.org/download.html 2018 ) . Another category of approaches aim to disjointly learn the features of each modality under cross-modality constraints such as cross-modal ranking ( Frome et al. , 2013 ; Lazaridou et al. , 2015 ; Kiros et al. , 2014 ) or feature distance ( Pan et al. , 2016 ; Xu et al. , 2015 ; Liong et al. , 2016 ) . Our approach belongs to this category and uses cross-correlation as cross-modality constraint . Crosscorrelation has been exploited to generate visual features attended by text for visual question answering ( Kim et al. , 2017 ; Yu et al. , 2017 ) . It has also been used to obtain cross-attention for few-shot learning ( Hou et al. , 2019 ) and image-text matching ( Lee et al. , 2018 ; Wei et al. , 2020 ) . In our work , we adopt the cross-correlation to generate both of audio and visual features attended by each other . The most similar to our cross-attention mechanism is the cross-attention module of Hou et al . ( 2019 ) , which computes cross-correlation spatially between features maps of two images ( sample and query ) . Whereas , our cross-attention is designed for video and is computed between two temporal sequences of different modalities . 3 METHODOLOGY . In this section , we introduce the proposed framework for weakly-supervised action and event localization . Fig . 1 illustrates the complete framework . We first present the multi-stage cross-attention mechanism to generate the audio-visual features in Sec . 3.1 . Then , we explain open-max classification to robustly distinguish the actions3 from unknown background in 3.2 . Finally , in Sec . 3.3 , we describe the training loss including two consistency losses designed to enforce temporal continuity of the actions and background . Problem statement : We suppose that a set of videos only with the corresponding video-level labels are given for training . For each video , we uniformly sample L non-overlapping snippets , and then extract the audio features U = ( ul ) Ll=1 ∈ Rdu×L with a pre-trained network , where ul is the du dimensional audio feature representation of the snippet l. Similarly , the snippet-wise visual features V = ( vl ) Ll=1 ∈ Rdv×L are also extracted . The video-level label is represented as c ∈ { 0 , 1 , . . . , C } , where C is the number of action classes and 0 denotes the background class . Starting from the audio and visual features , our approach learns to categorize each snippet into C + 1 classes and hence localizes actions in weakly-supervised manner . 3.1 MULTI-STAGE CROSS-ATTENTION MECHANISM . While multiple modalities can provide more information than a single one , the modality-specific information may be reduced while fusing different modalities . To reliably fuse the two modalities , 3For brevity we refer both action and event as action . we develop the multi-stage cross-attention mechanism where features are separately learned for each modality under constraints from the other modality . In this way , the learned features for each modality encodes the inter-modal information , while preserving the exclusive and meaningful intramodal characteristics . As illustrated in Fig . 1 , we first encode the input features U and V to Xu = ( xlu ) L l=1 and Xv = ( xlv ) L l=1 via the modality-specific fully-connected ( FC ) layers fu and fv , where x l u and x l v are in Rdx . After that , we compute the cross-correlation of Xu and Xv to measure inter-modal relevance . To reduce the gap of the heterogeneity between the two modalities , we use a learnable weight matrix W ∈ Rdx×dx and compute the cross-correlation as Λ = XTuWXv ( 1 ) where Λ ∈ RL×L . Note that xlu and xlv are l2-normalized before computing the cross-correlation . In the cross-correlation matrix , a high correlation coefficient means that the corresponding audio and visual snippet features are highly relevant . Accordingly , the lth column of Λ corresponds to the relevance of xlv to L audio snippet features . Based on this , we generate the cross-attention weights Au and Av by the column-wise soft-max of Λ and ΛT , respectively . Then , for each modality , the attention weights are used to re-weight the snippet features to make them more discriminative given the other modality . Formally , the attention-weighted features X̃u and X̃v are obtained by X̃u = XuAu and X̃v = XvAv . ( 2 ) Note that each modality guides the other one through the attention weights . This is to ensure the meaningful intra-modal information is well-preserved while applying the cross-attention . To extensively delve into cross-modal information , we repeatedly apply the cross-attention multiple times . However , during the multi-stage cross-attention , the original modality-specific characteristics may be over-suppressed . To prevent this , we adopt the dense skip connection ( Huang et al. , 2017 ) . More specifically , at stage t , we obtain the attended audio features by X ( t ) att , u = tanh ( t−1∑ i=0 X ( i ) att , u + X̃ ( t ) u ) ( 3 ) where X ( 0 ) att , u is Xu , and tanh ( · ) denotes the hyperbolic tangent activation function . Similar to X ( t ) att , u , the attended visual features X ( t ) att , v are generated for the visual modality . At the last stage te , we concatenate the attended audio and visual features to yield audio-visual features , Xatt = [ X ( te ) att , u ; X ( te ) att , v ] ( 4 ) where te is empirically set to 2 which will be discussed in the ablation studies in Section 4.3 . Discussion Applying the cross-attention ( Eq . 2 ) brings the audio and visual embeddings closer , while the skip connections ( Eq . 3 ) enforce modality specific information , more so with dense skip connections . Using both the cross-attention and the dense skip connections alternatively over multiple stages , we progressively learn optimal embeddings for fusion . Learning in this way , we aim to achieve right amount of compatibility between the two embeddings while preserving the modality specific information , in order to optimize for the training objective .

Summary: The authors propose to mix the independent embeddings of audio and visual data by a set of cross-attention layers to the task of audio-visual event localization. They weigh the features based on the correlation between the representations of both modalities to obtain the output representation. After several layers like these, combined with dense skip connections, the final output is a multimodal representation of the input where the features have been individually learned for each modality based on information from the other one. The final features are concatenated and passed through an open-max classifier. Finally, they design different losses to help enforce coherence and continuity among the proposed labels throughout the video.