Cross-Attentional Audio-Visual Fusion for Weakly-Supervised Action Localization
Authors: Jun-Tae Lee
Mihir Jain
Hyoungwoo Park
Sungrack Yun
Venue: ICLR 2021 Poster (2021)
Keywords: Audio-Visual
Multimodal Attention
Action localization
Event localization
Weak-supervision

Abstract: Temporally localizing actions in videos is one of the key components for video understanding. Learning from weakly-labeled data is seen as a potential solution towards avoiding expensive frame-level annotations. Different from other works which only depend on visual-modality, we propose to learn richer audiovisual representation for weakly-supervised action localization. First, we propose a multi-stage cross-attention mechanism to collaboratively fuse audio and visual features, which preserves the intra-modal characteristics. Second, to model both foreground and background frames, we construct an open-max classifier that treats the background class as an open-set. Third, for precise action localization, we design consistency losses to enforce temporal continuity for the action class prediction, and also help with foreground-prediction reliability. Extensive experiments on two publicly available video-datasets (AVE and ActivityNet1.2) show that the proposed method effectively fuses audio and visual modalities, and achieves the state-of-the-art results for weakly-supervised action localization.

Published as a conference paper at ICLR 2021
CROSS-ATTENTIONA. AUDIO-VISUAA FUSION FOR WEAKLY-SUPERVISE ACTION LOCALIZATION
Juntae Lee, Mihir Jain, Hyoungwoo Park & Sungrack Yun Qualcomm AI Research."
(juntlee, mi ja jaain, hwoopark, sungrack) qua comm. com
ABSTRACT
Temporally localizing actions in videos is one of the key components for video understanding. Learning from weakly-labeled data is seen as a potential solu- tion towards avoiding expensive frame-level annotations. Different from other works which only depend on visual-modality. we propose to learn richer audio- visual representation for weakly-superviie action localization. First, we propose a multi-stage cross-attention mechanism to collaboratively fuse audio and visual features, which preserves the intra-modal characteristics. Second, to model both foreground and background frames, we construct open-max classifier which treats the background class as an open-set. Third, for precise action localiza- tion, design consistency losses to enforce temporal continuity for the action- class prediction, and also help with foreground prediction reliability. Extensive experiments
foreground-predictton
publicly available video-datasets (AVE and ActivityNet show that the proposed method effectively fuses audio and visual modalities, and achieves the state-of-the-an results for weakly-supervised action localization.
INTRODUCTION
The goal of this paper is to temporally localize actions and events of interest in videos with weak- supervision. In the weakly-supervised setting, only video-level labels are available during the train- ing phase to avoid expensive and time-consuming frame-level annotation. This task is of great importance for video analytics and understanding. Several seeee--supervieed methods have been developed for it (Nguyen et al., 2018; Paul et al., 2018; Narayan et al., 2019; Shi et al., 2020; Jain et al., 2020) and considerable progress has been made. However, only visual information is ex- ploited for this task and audio modality has been mostly overlooked. Both, audio and visual data often depict actions from different viewpoints (Guo et al., 2019). Therefore, we propose to explore the joint audio-visual representation to improve the temporal action localization in videos.
A few existing works (Tian et al., 2018; Lin et al., 2019; Xuan et al., 2020) have attempted to fuse audio and visual modalities to localize audio-visual events. These methods have shown promising results, however, these audio-visual events are essentially actions that have strong audio cues, such as playing guitar, and dog barking. Whereas, we aim localize wider range of actions related to sports, exercises, eating etc. Such actions can also have weak audio aspect and/or can be devoid of informative audio (e.g. with unrelated background music). Therefore, it is key challenge to fuse audio and visual data in way that leverages the mutually complementary nature while maintaining the modality-specifii information.
In order to address this challenge, we propose novel multi-stage cross-attention mechanism. It progressively learns features from each modality over multiple stages. The inter-modal interaction is allowed at each stage only through cross-attention, and only at the last stage are the visually- aware audio features and audio-aware visual features concatenated. Thus, an audio-visual feature representation is obtained for each snippet in videos.
Separating background from actions/events a common problem in temporal localization. To this end, we also propose: (a) foreground reliability estimation and classification via open-max classifier and (b) temporal continuity losses. First, for each video snippet, an open-max classifier predicts
"Qullcomm AI Research initiative of Qualcomm Technologies, Inc.

---

Published as a conference paper at ICLR 2021
scores for action and background classes, which is composed of two parallel branches for action classification and foreground reliability estimation. Second, for precise action localization with weak supervision, design temporal consistency losses to enforee temporal continuity of action- class prediction and foreground reliability.
We demonstrate the effectiveness of the proposed method for weakly-supervised localization of both audio-visua events and actions. Extensive experiments are conducted on two video datasets for localizing audio-visual events (AVE') and actions (ActivityNet1.29) To the best of our knowledge, it is the first attempt to exploit audio-visual fusion for temporal localization of unconstrained actions in long videos.
2 RELATED WORK
Our work relates to the tasks of localizing of actions and events in videos, as well as the regime of multi-model representation learning.
Weakly-supervised action localization: Wang et al. (2017) and Nguyen al. (2018) employed multiple instance learning (Dietterich et al., 1997) along with attention mechanism to localize ac- tions in videos. Paul et al. (2018) introduced co-activity similarity loss that looks for similar temporal regions in a pair of videos containing a common action class. Narayan et al. (2019) pro- posed center loss for the discriminability of action categories at the global-level and counting loss for separability of instances at the local-level. To alleviate the confusion due to background (non- action) segments, Nguyen et al. (2019) developed the top-down class-guided attention to model background, and (Yu et al., 2019) exploited temporal relations among video segments. Jain et al. (2020) segmented a video into interpretable fragments, called ActionBytes, and used them effec- tively for action proposals. To distinguish action and context (near-action) snippets, Shi et al. (2020) designed the class-agnostic frame-wise probability conditioned on the attention using conditional variational auto-encoder. Luo et al. (2020) proposed expectation learning framework where the key instance is modeled as a hidden variable. All these works have explored various ways temporally differentiate action instances from the near-action background by exploiting only visual modality, whereas we additionally utilize audio modality for the same objective.
expectation-mmxxiiiztto maximization
Audio-visual event localization: The task of audio-visual event localization, as defined in the lit- erature, is classify each time-step into one of the event classes or background. This is different from action localization, where the goal is to determine the start and the end of each instance of the given action class. In (Tian et al., 2018), a network with mudio-guided attention was proposed, which showed prototypical results for audio-visua event localization, and cross mooos-tmd synchro- nized event localization. To utilize both global and local cues in event localization, Lin et al. (2019) conducted audio-visual fusion in both of video-level and snippet-level using multiple LSTMs. As- suming single event videos, Wu et al. (2019) detected the event-related snippet by matching the video-level feature of one modality with the snippet-level feature sequence of the other modality. Contrastingly, our cross-attention is over the temporal sequences from both the modalities and does not assume single-action videos. In order address the temporal inconsistency between audio and visual modalities, Xuan et al. (2020) devised the modality sentinel, which filters out the event- unrelated modalities. Encouraging results have been reported, however, the localization capability of these methods has been shown only for the short fixed-length videos with distinct audio cues. Differently, aim to fuse audio and visual modalities in order to also localize actions in long. untrimmed and unconstrained videos.
Deep multi-modal representation learning: Multi-modal representation learning methods aim to obtain powerful representation ability from multiple modalities (Guo et al., 2019). With the ad- vancement of deep-learning, many deep multi-mode representation learning approaches have been developed. Several methods fused features from different modalities in joint subspace by outer- product (Zadeh et al., 2017), bilinear pooling (Fukui et al., 2016), and statistical regularization (Aytar et al., 2017). The encoder-decoder framework has also been exploited for multi-modal learning for taage-to-image translation (Huang et al., 2018) and to produce musical translations (Mor et al.,
2

---

Published as a conference paper at ICLR 2021
fu
X
++
(1)
x2
y(a)
(phn)
w
W
Natt
g
V
+++
1 +
x
Open-max classifier
Multi-stage cross-attention
Figure I: The proposed architecture has two parts: modality fusion and open-max classification. (a) Fusion by multi-stage cross-attention The input audio (U) and visual (V) features are embedded by the two fully-connected layers fu and fu, and passed through the multiple stages of the cross- attention. At the Ith stage, the attended audio-visual embeddings, x((), and x are calculated using the results from the previous stages through dense skip connections, and activated by non- linear function. Here, ( and + denote concatenation and summation operations. This figure shows 2 2-stage case. The dense skip connections of two stages depicted as green and yellow arrows, respectively. At the last stage, two attended features are concatenated. (b) Open-max classifier takes the concatenated vudio-visual features input and generates classification scores for action classes and background. More detailed description is given in Appendix C.
2018). Another category of approaches aim to disjointly learn the features of each modality under cross-modality constraints such as cross-modal ranking (Frome et al., 2013; Lazaridou et al., 2015; Kiros et al., 2014) or feature distance (Pan et al., 2016; Xu ct al., 2015; Liong et al., 2016). Our approach belongs to this category and uses cross-correlation cross-modality constraint. Cross- correlation has been exploited to generate visual features attended by text for visual question an- swering (Kim et al., 2017; Yu al., 2017). It has also been used to obtain cross-attention for few-shot learning (Hou et al., 2019) and image-text matching (Lee et al., 2018; Wei et al., 2020). In work, we adopt the cross-correlation generate both of audio and visual features attended by each other. The most similar to our cross-attention mechanism is the cross-attention module of Hou et al. (2019), which computes cors-ccorelation spatially between features maps of two images (sample and query). Whereas, our cross-attention is designed for video and is computed between two temporal sequences of different modalities.
METHODOLOGY
In this section, we introduce the proposed framework for weakly-supervised action and event local- ization. Fig. illustrates the complete framework. We first present the multi-stage cross-altention mechanism to generate the audio-visual features in Sec. 3. Then, we explain open-max classifica- tion to robustly distinguish the actions? from unknown background in 3.2. Finally, in Sec. 3.3, describe the training loss including two consistency losses designed to enforce temporal continuity of the actions and background.
We suppose that a set of videos only with the corresponding video-level labels given for training. For each video, we uniformly sample L non-overlapping snippets, and then extract the audio features U = (u'){ E RduxL with a pre-trained network, where w' is the dn dimensional audio feature representation of the snippet I. Similarly, the snippet-wise visual features V (u')= E Edd, x.. are also extracted. The vidco-level label is represented as E {0, 1, where the number of action classes and 0 denotes the background class. Starting from the audio and visual features, our approach learns to categorize each snippet into C' + classes and hence localizes actions weakly-supervised manner.
Problem statement:
C),
3.1 MUL.TI-STAGE CROSS-ATTENTION MECHANISM
While multiple modalities can provide more information than single one, the modality sppecffcc information may be reduced while fusing different modalities. To reliably fuse the two modalities,
For brevity we refer hoth action and event action.

---

Published as a conference paper at ICLR 2021
develop the multi-stage cross-attention mechanism where features separately learned for each modality under constraints from the other modality. In this way, the learned features for each modality encodes the inter-modal information, while preserving the exclusive and meaningful intra- modal characteristics.
As illustrated in Fig. 1, we first encode the input features U and V to Xx (x/))/21 and X. (at))/1 via the modality-specifn
fully-connected (FC) layers fu and fu, where z and zy are in
compute the crrss-corrllation of Xu, and X, to measure inter-modal relevance. To reduce the gap of the heterogeneity between the two modalities, we use learnable weight matrix W E Rd, xd, and compute the cross-correlation
R After that,
A X WX,
where A E LXL, Note that o and xt, are 2 lz-normalized before computing the cross-correlation.
In the cross-correlation matrix, a high correlation coefficient means that the corresponding audio and visual snippet features are highly relevant. Accordingly, the lth column of A corresponds to the relevance of z L audio snippet features. Based on this, we generate the cross- rross-attention weights A, and A, by the column-wise soft-max of A and A?; respectively. Then, for each modality, the attention weights are used to re- re-weight the snippet features to make them more discriminative given the other modality. Formally, the attention- attention-weighted features X2 and X are obtained by
X,= X and X, = =p
(2)
Note that each modality guides the other through the attention weights. This is to ensure the meaningful intra-modal information is well-preserved while applying the cross-attention.
To extensively delve into cross-modal information, repeatedly apply the cross-attention multiple times. However, during the multi-stage cross-attention, the original modality-specifie characteristics may be sver-suppeessed. To prevent this, we adopt the dense skip connection (Huang et al., 2017). More specifically, at stage t, we obtain the attended audio features by
x att,a = tanh
x
(3)
att,u
1=0
where x- is X,, and tanh(.) denotes the hyperbolic tangent activation function. Similar to x(t) att the attended visual features x(t) are generated for the visual modality.
At the last stage ler we concatenate the attended audio and visual features to yield audio-visual
features,
Xatt = ( x((.)) ast.v
(4)
where 4g empirically set which will be discussed the ablation studies Section 4.3.
Discussion Applying the cross-attention (Eq. 2) brings the audio and visual embeddings closer, while the skip connections (Eq. 3) enforce modality specific information, more so with dense skip connections. Using both the cross-attention and the dense skip connections alternatively over mul- tiple stages, we progressively learn optimal embeddings for fusion. Learning in this way, we aim to achieve right amount of compatibility between the two embeddings while preserving the modality specific information, in order to optimize for the training objective.
3.2 OPEN-MAX CLASSIFICATION
Video segments can be dichotomized into foreground actions and background. For precise action localization, distinguishing the background from the actions is important as well as categorizing the action classes. However, unlike action classes, the background class comprises extremely diverse types of non-actions. Therefore, it is not possible train for the wide range of background classes that the model may confront at the test time.
To resolve this problem, address the background Boult, 2016). As illustrated in Fig. 1, we construct an open-max classifier on top of the multi- stage cross-attentional feature fusion. Specifically, the open-max classifier consists of parallel
open set (Dietterich, 2017; Bendale &
4

---

Published as a conference paper at ICLR 2021
(a)
(b)
(c)
Video with 'paintball' action
Vidco with 'kayaking' action
Figure 2: Visualization of class activation sequences for the target actions in two example videos: The ground-truth segments are shown in (a). The class activation sequences obtained without Ccont and Cpecu are shown in (b), which improve and get better aligned the ground-truth segments when these continuity losses are used as shown in (c). The activation is depicted in gray-scale, where lower intensity indicates more strong activation.
FC layers for action classification and foreground reliability estimation. The attended audio-visus visual feature Watt. where = 1..... L, is fed snippet-wise into the open-max classifier. The first FC layer outputs snipper-wise activation vector = [ ,,,((, for action classes, which is converted to probability scores, Pue by soft-max function.
Simultaneously, the second FC layer is applied on =uat, followed by a sigmoid function to estimate its foreground reliability, ze [0, 1]. The foreground reliability, j², is the probability of snippet I belonging to any action class. The low reliability indicates that no action occurs in the snippet. Therefore, we compute the probability for the background class as the complement of μ², by Pig 1-
Lastly, the open-max classifier outputs the probability distribution yf over C classes including the background and C actions as
p' = PPggi p p'P'
(5)
3.3 TRAINING LOSS
Next, we describe the loss functions to train our model. The actions or foreground do not abruptly change over time. To impose this constraint, we devise two types of temporal continuity losses.
Foreground continuity loss: Foreground continuity implics two important properties for neighbor- ing snippets: (a) similar foreground reliability in class-agnostic way, and (b) consistent open-max probabilities for a target foreground class.
The first of the two constraints is imposed via alass-agnostic foreground continuity:
B/2
piug
Σ G(1)
(6)
B+1
B/2
where G(i) is a Gaussian window of width B + 1 apply temporal smoothing over foreground liability around Ith snippet. For the second constraint, temporal Gaussian smoothing is applied over open-max probability of video-level ground- truth action class, è, to obtain class-specific foreground continuity:
B/2 Σ
MPp B+1
G(1)
B/2
Finally, the foreground continuity loss is defined as:
Lcout
1
Whyg + 1p2 Mapl-
(8)
L
The foreground continuity loss imposes temporal continuity of foreground, and hence also helps in separating background from the action classes.
5

---

Published as a conference paper at ICLR 2021
Table 1: Ablation for multi-stage cross-attention. The results for different stages of the cross- attention are reported for the AVE and ActivityNet1.2 datasets. The comparison with the uni-modal approach shows the impact of leveraging the multi-modality and the cross-attention.
Uni-moda
Multi-modas
Audio
Visual
O-stage
1-stage
2-stage
3-stage
AVE
Accuracy (%)
32.1
45.2
65.0
75.0
77.1
75.6
0.5
12.3
38.3
37.6
42.1
44.8
39.5
10.9
32.9
32.4
37.8
33.8
ActivityNet1.2
mAP@IoU (%)
0.7
9.7
25.4
26.7
29.5
30.8
27.9
0.8
7.6
19.2
19.4
20.8
22.5
20.9
Avg.
7.8
22.1
22.0
24.1
26.0
23.3
Pseudo localization loss: Here, we consider the action or background class continuity, which im- plies that the open-max probabilities, p', agrees with the classification of neighbouring snippets. This can be used to obtain the pseudo label for snippet (. We first average the open-max prediction of N neighbor snippets and itself, y N+I N Ei LiiiN We set èt arg (a as the pseudo label, but only retain when the largest class probability of o' higher than a predefined threshold Accordingly, the pseudo localization loss is formulated by
C peee
Σ (max((d) 2 r) log p' pce))
L.
Total loss: Additionally, we employ the multiple instance learning (MIL) and co-activity similarity (CAS) losses (Paul et al., 2018). The final loss L is defined by
Cmil + aLcas + 3Loomt + 7
(10)
where Luil and Leas denote MIL and CAS losses, respectively. For details Appendix D.
Figs. 2 (b) and (c) compare the class activation sequences along the temporal axis for the target classes between the models trained without and with the two consistency losses, respectively. We see that class activations are more continuous in the model with the consistency losses.
EXPERIMENTS
In this section, we provide experimental analysis and comparative evaluation to show the effective- ness of the proposed method. More experiments and qualitative results are in the Appendix.
4.1 DATASETS AND EVALUATION METHOD
Datasets: We evaluate our approach on Andio-Visual Event (AVE) and ActivityNet1 1. datasets.
AVE dataset is constructed for audio-visual event localization, which contains 3,339 training and 804 testing videos, each lasting 10 seconds with event annotation per second. There are 28 audio- visual event categories covering wide range of domains, such as animal and human actions, vehicle sounds, and music performance. Each event category has both audio and visual aspects, e.g. church bell, baby crying, man speaking etc.
ActivityNetI.2 2 a temporal action localization dataset with 4,819 train and 2,383 validation videos, which in the literature is used for evaluation. It has 100 action classes of wider variety than AVE dataset, with on an average 1.5 instances per video. The average length the videos in this dataset is 115 seconds, often with weak audio cues, which makes action localization well leveraging audio harder.
Evaluation metrie: We follow the standard evaluation protocol of each dataset. For the AVE dataset, we report snippet-wise event prediction accuracy. For the ActivityNetl. dataset, we gen- erate the action segments (start and end time) from snippet-wise prediction (details are described in the following section), and then measure mean average precision (mAP) at different intersection union (loU) thresholds.

---

Published as a conference paper at ICLR 2021
Table Ablations for the consistency losses and open-max classifier. Consisteney losses: The lower part of the table shows the impact of each of the two consistency losses, when used with the open-max classifier. Open-max vs soft-max: The results for the soft-max are also shown, which demonstrates the advantage of foreground/backgromm modelling by the open-max classification on both the datasets. The model with 2- stag cross-attention is used.
AVE
ActivityNetl 22 [mAP@ IoU (%)
Method
Coont
Lpeen
Accuracy (%)
0.5
0.6
0.7
0.8
Avg.
Soft-Max
S-I
68.5
39.4
35.7
30.7
19.8
23.8
0-1
64.9
39,9
33.7
23.8
14.3
20.3
Open-Max
O-Il
75.9
44.1
37.4
31.1
22.4
25.7
O-1II
77.1
44.8
37.8
30.8
22.5
26.0
(a)
(d)
(e)
Figure 3: Visualization of the action localization result for an example video from Activity Net1 .2. The ground truth is shown in (a). highlighted in green. The localization and the class activation sequence of the visual-only model are shown in (b) and (c), respectively. Finally, the localization and the class activation sequence for the proposed audio-visual method are shown in (d) and (e).
4.2 FEATURE EXTRACTION AND IMPLEMENTATION DETAILS
Feature extraction: We use the 13D network (Carreira & Zisserman, 2017) and the ResNet152 architecture (He et al., 2016) to extract the visual features for ActivityNet1.2 and AVE, respec- tively. The 13D network is pre-trained on Kinetics-400 (Kay et al., 2017), and the features consist of two components: RGB and optical flow. The ResNet 152 is pre-trained on the ImageNet (Rus- sakovsky al., 2015), and the features extracted from the last global pooling layer. To extract the audio features, we use the VGG-like network (Hershey et al., 2017), which pre-trained on the AudioSet (Gemmeke al., 2017), for both AVE and ActivityNet1. .2 datasets.
Implementation Details:
We set d₂ to 1,024, and the LeakyRelu and hyperbolic tangent functions respectively used for the activation of modality-
layers and cross attennttnnnionn modules. In training, the parameters are initialized with Xavier method (Glorot & Bengio, 2010) and updated by Adam optimizer (Kingma & Ba, 2015) with the learning rate of 10-4 and the batch size of 30. Also, the dropout with ratio of 0.7 is applied for the final attended audio- visual features. In the loss, the hyper parameters are set as
4, a 0.8, B 0.8 and 7
Localization at test time: For event localization at test time, i.c. snippet classification, each snippet 1 classified into one of event classes (including background) by arg max p' (c), where p' is the open-max probability of snippet 1. For action localization, we follow the two-stage thresholding scheme (Paul et al., 2018). The first threshold applied to filter out the classes that have video- level scores less than the average over all the classes. The second threshold is applied along the temporal axis to obtain the start and end of each action instance.
4.3 ABLATION ANALYSIS
evaluate the effectiveness of the multi-stage cross-attention in audio-visual fusion, compare two uni-modal methods (audio visual) and four multi-modal methods with different stages (0-3 stages) on the AVE and ActivityNetl.2 datasets Table 1. The pseudo-label losses and the open-man classifiers are used in all six cases. In the uni-modal methods, the input feature is embedded using an FC layer, and then fed into the open-max classifier. The O-stage method denotes naive fusion, where audio and visual features fused by simple
Multi-stage cross-attention:
7

---

Published as a conference paper at ICLR 2021
Table 3: Impact of dense skip con- nections: Ablation studies on dense skip connection in terms of average of for the Activi- tyNet1 dataset. For 2-stage model, no, skip, and dense connections are verified.
Table Comparison of the number of FLOPS and the average mAP@ 10.:00000000::09955 on the ActivityNet1.2 dataset for visual-only, 1 1-stage, and 2-stage models. d x d are the dimensions for the cross cooss-correlation matrix W.
Method
dz
No. FLOPS
Avg. mAP
Visual-only
1024
2.3x10
22.1
Method
Avg. mAP
I-stage
1024
3,5x 10
24.1
w/o skip connection
24.1
2-stage
1024
4.0x10"
26.0
w/ skip connection
24.9
Visual-only
512
1.2x10
21.0
dense skip connection
26.0
2-stage
512
1.7x10"
25.9
catenation. Even this naive fusion yields higher performance than the uni-modal methods on the AVE dataset. However, that is not the case with more challenging task of the action localization on ActivityNetl.2 .2 dataset. Furthermore, all the later stages improve considerably over O-stage and the uni-modal cases, for the both datasets. The 2-stage cross-attention achieves the best performance for the both datasets (more in Appendix A). Interestingly, even with the minimal audio cue in Ac- tivity 2 (avg. mAP of audio only is 7. 7.8%), the proposed audio-visual features improve the avg. mAP over visual-only and naive fusion (0-stage) models by 4%.
Fig. 3 shows the qualitative results of the proposed and visual-only models given an example of the ActivityNetl.2 .2 dataset. At the beginning of the video, a performer is shown without any activity. The visual-only model incorrectly predicts the beginning part as a target action while our proposed model correctly predicts it as background. Also, the visual-only model cannot catch the action at the last part of the video since it is visually similar across the frames and has minimal visual activity. Whereas, our model correctly recognizes the last part as an action, owing to the multi-stage cross- attention of effective fusion of the two modalities. More qualitative results are in Appendix E.
We show the ablation over the two proposed losses, Coont and Lpeeu while using Open-Max classifier and 2-stage cross-altention, in the lower part of the Table 2. We denote the method with only Lromt loss by O-1 and with only Eysen loss by O-II. The proposed method (O- III) with both of the losses performs the best suggesting the importance of both of the losses. Further, O-II outperforms O-I by big margin on both the datasets, implying that the pseudo localization loss more critical for the action localization (more in Appendix B. This result demonstrates that guiding temporal continuity is essential in the long untrimmed videos as well as the short ones.
Consistency losses:
Open-max classifier: We compare the open-max classifier with the soft-max classifier where the last FC layer outputs activations for C + I classes are normalized by the soft-max function. As the background considered closed set the soft-max approach, the foreground continuity loss not available. The soft-max is denoted by S-I in Table 2. Both O-1I and O-1II versions of the open-max outperform the S-I method with the soft-max. The O-III method improves the accuracy by 8.6% on the AVE dataset and the avg. mAP by 2.2% on the ActivityNet dataset. For further analysis Appendix B.2. This shows the advantage of modelling background with the open-max classifier.
Dense skip connections: We evaluate the impact of dense skip connections in Table 3 for 2-stage model on the ActivityNet1.2. Compared to no skip connection, performance is improved with the skip connections, and further boosted with the dense skip connection avg. mAP of 26.0%. This shows preserving the modality specific information leads to better fusion and action localization.
4.4 MODEL EFFICIENCY
Though we successfully leverage the audio modality to improve action localization performance, the added modality leads to increased computational cost. The trade-off between efficiency and performance due to the fusion with audio modality is demonstrated Table 4. When using feature dimension, d, 1024, the fusion increases the computation over visual-only method by about 52% and 74% after 1-stage and 2-stage, respectively. When we reduce d₂ to 512, the visual-only model gets affected while the 2-stage model maintains its performance 25.9%. Thanks to the effective- ness of the proposed fusion, even with smaller d, its avg. mAP is well above that of video-only model with d= 1024, while using about 26% less computation (1.7 MFLOPS vs 2.3 MFLOPS).
8

---

Published as a conference paper at ICLR 2021
Table 5: Comparison of the proposed method with the state-of-the-art fully and weakly-superrised methods (separated by 'f") on the AVE dataset. Snippet-level accuracy (%) is reported.
Method
Tian al. (2018)
Lin et al. (2019) Owens & Efros (2018) Xuan al. (2020)
Proposed
Accuracy (%)
73.1
75.4 / 74.2
72.3 68.8
77.1 / 75.7
-/ 77.1
Table 6: Comparison of our method with the state-of-the-ar action localization methods on the ActivityNetl .2 dataset. The mAPs (%) different IoU thresholds and average mAP across the IoU thresholds are reported. indicates audio-visual models. *experiment done using author's 's code.
Method
Supervision
IoP@ (%)
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
Avg.
Zhao et al. (2017)
Full
41.3
38.8
35.9
32.9
30.4
27.0
22.2
18.2
13.2
6.1
26.6
Paul et al. (2018)
Weak
37.0
33.5
30.4
25.7
16.6
12.7
10.0
7.0
4.2
1.5
18.0
Liu et ul. (2019b)
Weak
37.1
33.4
29.9
26.7
23.4
20.3
17.2
13.9
9.2
5.0
21.6
Liu al. (2019s)
Weak
36.8
22.0
5.6
22.4
Jain et (2020)
Weak
39.4
15.4
Shi et al. (2020)
Weak
41.0
37.5
33.5
30.1
26.9
23,5
19.8
15.5
10.8
5,3
24.4
Luo et al. (2020)
Weak
37.4
23.1
2.0
20.3
Tian et al. (2018)'
Weak
15.4
13.9
12.5
11.2
10.2
9.1
7.6
5.7
1.6
0.3
8.8
Naive fusion (O-stage)
Weak
41.2
38.4
34.8
31.8
26.3
17.0
5.6
2.2
0.8
0.2
19.8
Naive fusion (0-stage)) CL
Weak
39.4
37.0
33.5
30.6
27.7
23.6
20.0
13.6
3.0
0.6
22.9
Ours!
Weak
44.8
42.1
37.8
34.2
30.8
26.7
22.5
15,9
4.0
1.0
26.0
Ours! (efficient)
Weak
45.0
41.8
38.3
34.0
29.7
26.3
22.1
15.7
1.0
25.9
4.5 COMPARISON WITH THE STATE-OF-THE-ART
Audio-visual event localization: Table we compare the proposed method with the recent fully and
methods the AVE dataset for audio-visur event localization task. In the weakly-supervised setting, our method performs better than all of the cxisting methods at least by 1.4%. Note that, even though learned in weak-supervision, accuracy (77.1%) %) to the fully-supervised accuracy of the state-of-the-an method (Xuan et al., 2020).
approach achieves comparable
Temporal action localization: In Table 6, apply the proposed method to weakly-supervin action localization long duration videos of the ActivityNetl.2 dataset. We report results for our method well its efficient version from Section 4.4. The mAP scores varying IoU thresholds are compared with the current state-of-the-aa methods. Both our method and its efficient version achieve the highest mAPs for 8 out of 10 IoU thresholds, and outperform all of the previous methods with the avg. mAp of 26.0%. We also significantly outperform the audio-visual based method of Tian et al. (2018) by the avg. mAP of 17.2%. Additionally, we compare with two naive fusions without the cross-attedd -ttsstantention (O-stage, SoftMax) with and without the continuity losses (denoted as CL the Table), both are bettered comfortably by our method. This demonstrates that the effective fusion of audio and visual modalities is critical for action localization. Furthermore, our approach even comparable to the fully-supervised method (Zhao et al., 2017).
5 CONCLUSION
We presented a novel approach for weakly-supervised temporal action localization in videos. In contrast to other methods, we leveraged both audio and visual modalities for this task. This is the first attempt at audio-visual localization of unconstrained actions in long videos. To collaboratively fuse audio and visual features, we developed the multi-stage cross-attention mechanism that also preserves the characteristics specific to each modality. We proposed to use the open-max classifier to model the action foreground and background, in absence of temporal annotations. Our model learss to classify video snippets via two consistency losses that enforce continuity for foreground reliability and open-max max probabilities for action classes and the background. We conducted exten- sive experiments to analyze each of the proposed components and demonstrate their importance. Our method outperforms the state-of-the-art results both AVE and ActivityNetl. . datasets.

---

Published as a conference paper at ICLR 2021
REFERENCES
Yusuf Aytar, Lluis Castrejon, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Cross-modal scene networks. In IEEE Trans. Pattern Anal. Mach. Intell., volume 40, pp. 2303-2314, 2017. 2
Abhijit Bendale and Terrance E. Boult. Towards open deep networks. In CVPR. 2016. 4
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics
dataset. In CVPR. 2017. 7
Thomas G. Dietterich. Steps toward robust artificial intelligence. An Mag., volume 38, pp. 3-24. 2017. 4
Thomas G. Dietterich, Richard H. Lathrop, and Tomás Lozano-Pérez Pérez. Solving the multiple instance problem with axis-parallel rectangles. In Artif. Intell., volume 89, pp. 31-71. 1997.
Andrea Frome, Greg S. Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. DeVise: A deep visual-semantic embedding model. In NIPS. 2013. 3
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In arXiv preprint arXiv: /6/.6/66660..77 2016. 2
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP. 2017. 7
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS. 2010. 7
Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep multimodal representation learning: A survey. In IEEE Access, volume 7, pp. 63373-63394, 2019. 1,2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR. 2016.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, al. CNN architectures for large-scale audio classification. In ICASSP. 2017.
Ruibing Hou, Hong Chang, MA Bingpeng, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In NeurIPS. 2019.
Gao Huang. Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR. 2017. 4
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In ECCV. 2018.
Mihir Jain, Amir Ghodrati, and Cees G. M. Snoek. ActionBytes: Learning from trimmed videos to localize actions. In CVPR. 2020. 1, 2,
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. In arXiv preprint arXiv: 170.006550000 2017.
Jin-Hwa Kim, Kyoung- Woon On, Woosang Lim, Jeonghee Kim, Jung- Woo Ha, and Byoung-Tak Zhang. Hadamard product for low-rank bilinear pooling. In ICLR. 2017. 3
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR. 2015.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. In NIPS. 2014.
Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. Combining language and vision with a multimodal skip-gram model. In HI.T-NACIL. 2015.
10

---

Published as a conference paper at ICLR 2021
Kuang- Huen Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV. 2018.
Y. Lin, Y. Li, and Y. F. Wang. Dual-modality seq2seq network for audio-visual event localization.
In ICASSP. 2019. 2,
Venice Erin Liong, Jiwen Lu. Yap-Peng Tan, and Jie Zhou. Deep coupled metric learning for cross- modal matching. In IEEE Trans. Multimedia, volume 19, pp. 1234-1244. 2016. 3
Daochang Liu, Tingting Jiang, and Yizhou Wang. Completeness modeling and context separation for weakly supervised temporal action localization. In CVPR. 2019a.
Ziyi Liu, Le Wang, Qilin Zhang, Zhanning Gao, Zhenxing Niu, Nanning Zheng, and Gang Hua. Weakly supervised temporal action localization through contrast based evaluation networks. In ICCV. 2019b.
Zhekun Luo, Devin Guillory, Baifeng Shi, Wei Ke, Fang Wan, Trevor Darrell, and Huijuan Xu. action localization with pnpecttaion-maximiznen multi-instance learning. In ECCV. 2020. 2, 9
Noam Mor, Lior Wolf, Adam Polyak, and Yaniv Taigman. A universal music translation network. In arXiv preprint 1805.0788877888 2018.
Sanath Narayan, Hisham Cholakkal, Fahad Shahbaz Khan, and Ling Shao. 3C-Net: Category count and center loss for beakly-supervised action localization. In ICCV. 2019. 1,2
Phuc Nguyen, Ting Liu, Gautam Prasad, and Bohyung Han. Weakly supervised action localization by sparse temporal pooling network. In CVPR. 2018. 1,2
Phuc Xuan Nguyen, Deva Ramanan, and Charless C. Fowlkes. Weakly-supervised action localiza- tion with background modeling. In /CCV. 2019. 2
Andrew Owens and Alexei A. Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV. 2018. 9
Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embedding and translation bridge video and language. In CVPR. 2016.
Sujoy Paul, Sourya Roy, and Amit K. Joy-Chowdhury. W-TALC: activity localization and classification. In ECCV. 2018. 1, 6, 9, 15
temporal
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. In Ini. J. Comput. Vis., volume 115, pp. 211-252. 2015. 7
Baifeng Shi, Qi Dai, Yadong Mu, and Jingdong Wang. Weakly-supervised action localization by generative attention modeling. In CVPR. 2020. 1,2,9
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In ECCV. 2018. 1, 1,
Limin Wang, Yuanjun Xiong, Dahua Lin, and Lue Van Gool. UntrimmedNets for weakly supervised
action recognition and detection. In CVPR. 2017. 2
Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, and Feng Wu. Multi-modality cross attention network for image and sentence matching. In CVPR. 2020. 3
Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. Dual attention matching for audio-visual event local- ization. In ICCV, 2019. 2
Ran Xu, Caiming Xiong, Wei Chen, and Jason J Corso. Jointly modeling deep video and composi- tional text to bridge vision and language a unified framework. In AAAI. 2015. 3
Hanyu Xuan, Zhenyu Zhang, Shuo Chen, Jian Yang, and Yan Yan. Cross-modal attention network for temporal inconsistent audio-visual event localization. In AAAI 2020. 1, 9
11

---

Published as a conference paper at ICLR 2021
Tan Yu, Zhou Ren, Yuncheng Li, Enxu Yan, Ning Xu, and Junsong Yuan. Temporal structure mining for weakly supervised action detection. In ICCV. 2019. 2
Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In ICCV. 2017.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. In EMNLP. 2017.
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. ICCV, 2017. 9
12

---

Published as a conference paper at ICLR 2021
A ANALYSIS ON MULTI-STAGE CROSS-ATTENTION
In this section, we conduct extensive analysis for the impact of the multiple stages and dense skip connection of the proposed cross-attention mechanism. Tables 7 and show the experimental re- sults.
Training multiple stages of cross-attention: As shown in the Table 1, the 3-stage model suffers from performance drop. analyze this, in Table 7, we compare 2- and 3-stage models on each of 'w/o skip connection', "w/ skip connection', and "w/ dense skip connection'. Without the skip connection, 3-stage model improves over 2-stage model, which intuitively expected. With the skip connection, avg. mAP of 3-stage model drops compared to 2-stage model. from 24.9% to 23.2%. But, when the third stage is appended to the trained (and now frozen) stages of 2-stage model, the avg. mAP maintained 24.9%. Similarly, with the dense skip connection, training the entire 3-stage model end-to-end leads to degraded performance. But, when training the model frozen till the second stage the drop much less. The fact that, in 3-stage model, better performance is obtained when training with first two stages frozen compared to training end-to-end, shows that the optimization gets hard in the latter. Therefore, we conclude that though the third stage helps without the skip connections, due to harder optimization with more stages and (dense) skip connections, 2-stage model the optimal choice.
Table 7: Comparing 2-stage and 3-stage models: Training connections in comparison to the 2-stage model.
of 3-stage model is analyzed with skip
Method
Model
Avg. mAP
2-stage
24.1
w/o skip connection
3- stsge
24.7
2- stage
24.9
w/ skip connection
3 stage
23.2
3 stage (frozen till 2nd stage)
24.9
2-stage
26.0
w/ dense skip connections
3-stage
23.3
3-stage (frozen till 2nd stage)
25.2
Table 8: Need for multi-stage: Ablation study on the size of the dimension for the cross-correlation matrix W E [dd xdr, for 1-stage model on the AVE dataset. 2-stage model with W e 1024x is the proposed.
Method
dy
dg,
Accuracy($$)
128
72.7
128
1024
73.2
128
2048
73.7
512
512
74.8
512
1024
74.8
512
2048
74.7
1-stage
1024
512
73.2
1024
1024
75.0
1024
2048
75.0
2048
512
73.7
2048
1024
74.3
2048
2048
74.4
2-stage
1024
1024
77.1
Need for multi-stage cross-attention: In Table 8, we experiment with 1-stage model, varying the size of dimensions (dr.u and dr,,)) of the cross-cores cross-correlation matrix W on the AVE dataset. We tried several hyper-parameter settings in 1-stage model, but of them could outperform the default
13

---

Published as a conference paper at ICLR 2021
setting (dz,u 1024, dz,0 1024) of 2-stage model with more parameters. Instead of increasing the parameters in 1-stage model itself, when an additional stage is added (i.e. a weight matrix learned with non-linear activation function) better performance is achieved. Indeed, it is often not trivial to replace a sequence of non-linear functions with another non-linear function as we experimentally observe here. The intention behind the multi-stage is also to extensively delve into cross-modal information, progressively learning the embeddings for each modality.
B ANALYSIS FOR CONSISTENCY LOSSES AND OPEN-MAX CLASSIFICATION
B. ANALYSIS OF CONSISTENCY LOSSES ON DIFFERENT STAGE MODELS
In Table 9, we conduct the analysis for the consistency losses for 0, and 3-stage models as well as the chosen 2-stage model.
Effect of losses on different stage models: The impact of continuity losses is analogous on 1-, 2- and 3-stage models. Each of the two continuity losses help, but the pseudo localization loss (Cpseu) ) is more effective. Also, there is further benefit of using them together for almost all the IoU thresholds and stages. O-stage model, iee. without the cross-attention, O-II shows the highest snippet-level performance the AVE dataset, but the lowest temporal action localization performance the ActivityNetl .2 dataset. From this, we understand that Lpson has difficulty achieving continuity when audio and visual features are overly heterogeneous. Consequently, clear benefit observed when the cross-attention is used.
Interdependence of cross-attention and pseudo localization loss: all 0-3 stage models, marginal, and the pseudo localization critical to the performance. This follows from Eq. 9, where Cpee only activated at snippet when classification over its neighboring snippets does not strongly agree on the action class or background. To analyze this, we check how frequently Lpsou is activated when cross-altention is not used and when it is used. For O-stage model, without continuity losses, Eeeen is activated on 1 11.1% % snippets of the ActivityNetl. training set. The same frequency is 38.2% for 2tstage model, again without the continuity losses. This shows that when the cross- cooss-attention is used, more often the open-max classification of snippet fails to strongly agree with its neighbors. Therefore, the pseudo localization loss is much needed to enforce the continuity.
When comparing the O-I of that the performance improvement by stacking the cross-attention is
Table 9: For 0-3 stage models, ablation analysis consistency losses for open-max (O-0, O-I, O-II, and O-III) classifiers on the AVE and the ActivityNet1.2 datasets. O-IlI of 2-stage is the proposed.
AVE
ActivityNetl .2 [mAP@ loU (%)
Method
Lcont
Lpeen
Accuracy (%)
0.5
0.6
0.7
0.8
Avg.
0-1
57.7
42.8
35.9
27.6
7.2
21.0
O-stage
O-II
66.8
29.4
25.3
20.5
13.6
16.5
O-III
d
65.0
37.6
32.4
26.7
19.4
22.0
O-I
64.6
39.8
33.4
27.4
13.0
21.9
I-stage
0-I1
73.7
41.5
34.9
28.4
20.9
24.0
O-III
75.0
42.1
35.3
29.5
20.8
24.1
0-1
64.9
39,9
33.7
23.8
14.3
20.3
2tstage
O-11
75.9
44.1
37.4
31.1
22.4
25.7
O-1II
77.1
44.8
37.8
30.8
22.5
26.0
0-1
66.2
38.4
31.8
25.5
17.7
21.5
3 3-stage
O-11
74.3
39.8
33.7
27.9
20.7
23.2
O-III
74.6
39.5
33.8
27.9
20.9
23.3
B.2 ANALYSIS LOSSES AND OPEN- MAX CLASSIFICATION ON 2-STAGE MODEL
In Table 10, conduct more extensive analysis for the consistency losses and the open-max clas- sifier. Specifically, we replace the open-max classification approach with soft-max one. Then, for both classifiers with the 2-stage cross-attention, ablate the foreground continuity or pseudo local-
14

---

Published as a conference paper at ICLR 2021
ization losses where CAS and MIL losses commonly used. First, the performance gap between S-O and O-O, where only CAS and MIL losses are used, shows the difficulty of learning two parallel branches in weakly-supervised manner. However, when adding the pseudo localization loss, (S-I and O-II), the open-max classification approach further improved than the soft-max. Hence, the pseudo labels reduce the fallacious action classification of snippets and open-set background modeling than the closed-set modeling.
effective the
Next, O-I and O-II shows higher performance than O-0. Similarly, S-I is superior S-O. This indicates that erroneous classifications are suppressed by the correctly classified neighbors when using the consistency losses. Also, comparing O-I and O-lI, the pseudo localization loss gives more performance improvement. This is because the pseudo localization loss addresses the consistency of classification scores of all the classes including background, while the foreground continuity loss smoothens foreground reliability being class-agnostic or only for the target class. For all of the loU thresholds (except 0.7), O-III, open-max classification with both of the consistency losses, yields the highest performance. Therefore, all of the proposed open-max classification and consistency losses are effective to temporal action or event localization videos.
Table 10: Ablation analysis of 2-stage model on consistency losses for soft-max (S-0 and S-I) and open-max (O-0, O-I, O-II, and O-IIII classifiers on the AVE and the ActivityNet1.2 datasets. O-III of is the proposed.
AVE
ActivityNetL.2 [mAP@ loU (%)
Method
Lcont
Lpeen
Accuracy (%)
0.5
0.6
0.7
0.8
Avg.
S-0
62.1
36.4
28.4
22.7
15.8
19.6
S-I
68.5
39.4
35.7
30.7
19.8
23.8
O-0
35.4
27.5
22.9
12.7
18.7
0-1
64.9
39.9
33.7
23.8
14.3
20.3
O-11
75.9
44.1
37.4
31.1
22.4
25.7
O-III
77.1
44.8
37.8
30.8
22.5
26.0
C DETAILS OF THE PROPOSED ARCHITECTURE
Layer/Oppeation
No. parameters
Input
Output
LeakyRelu
d,
X. Rd
fo LeakyRelu
d. x dz
X. Rd
X.. X.
X Rdxx X!") gg
Dense Connection Tanh
X.. X.,. _ X0
x E g x() e R2.x2
((())
d, x dz
(1) x(i)
X gdxx X! CR
Dense Connection Tanh
X,, X., ((U)
un X 813 X
x
RR.XL x e RR
Concatenation
x X
X., E xL
fu Soft-Max
2d,
Xm
fu Sigmoid
2d, x
(1 g
D MULTIPLE INSTANCE LOSS AND CO-ACTIVITY SIMILARITY LOSS
We apply multiple-instance learning loss for classification. The prediction score corresponding class is computed as the average of its top & activations over the temporal dimension. Co-activity similarity loss (CASL) (Paul al., 2018) is computed over two snippet sequences from a pair of videos, to have higher similarity when the videos have a common class.
E QUALITATIVE EVALUATION
We provide additional qualitative results for action localization on the ActivityNet1.2 dataset. Fig. compares the proposed method with the method trained visual modality (" ("Visual-only'). The
15

---

Published as a conference paper at ICLR 2021
Ground-truth
Visual-only
Proposed
(a) Cheerleading
Ground-truth
Visual-only
Proposed
(b) Playing violin
Ground-truth
Visual-only
Proposed
Grooming horse
Ground-truth
Visual-only
Proposed
(d) Playing field hockey
Figure 4: Qualitative results for action localization. Ground-truth (green), prediction by the visual- only method (orange), and prediction by the proposed method (blue) are shown. Class activation sequences are visualized below each prediction, darker shade means higher activation.
open-max classifier and total loss function are commonly used for both. In Figs. 4(a) and (b), be- cause the videos static in visual modality, the background segments in early parts of videos are miss-localized as actions in the visual-only model. Contrarily, proposed method distinguishes the background based on the action-related audio (cheerleading music and violin sound). In Fig. 4(c), the brushing sound is overlapped with the loud human narration lasting throughout videos. Nev- ertheless, proposed method effectively extracts the crucial audio cues and fuses them with the visual ones. In Fig. 4(d), even though the early part of the action is visually occluded by large logos, our method exactly localizes the action. Also, for all of the class activation sequences, the activations by the proposed method are more consistently high for actions. This means that collaboration of audio and visual modalities is more robust distinguishing foreground from background.
Fig. illustrates the cases where audio degrades the performance. Fig. (a) shows example video for action class 'playing violin'. The violin sound of the soloist and the band is intermingled in the video. In the end, the sound of violin continues making model predict the action but since camera focuses on the band, the ground-truth does not include those frames. Fig. (b) shows an example of action 'using parallel bars'. Here the repeated background music is irrelevant action,
16

---

Published as a conference paper at ICLR 2021
Ground-truth
Visual-only
Proposed
(a) Due to ground-truth labelling confined to visual modality
Ground-truth
Visual-only
Proposed
(b) Due to background music repeated for a long time
Figure 5: Examples where localization performance is degraded by audio.
therefore the class activation off in the last part. However, thanks to the visual modality, prediction is still reasonable.
17


Reviews:
[{"review_id": "cR-vwQghi_r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to learn richer audio-visual representations for weakly-supervised action localization. First, the authors propose a multi-stage cross-attention mechanism to collaboratively fuse audio and visual features. Extensive experiments on two publicly available video-datasets (AVE and ActivityNet1.2) show that the proposed method appears to be effective.\nThe authors did not give the explanation of the built network in more details. Moreover, the motivation of the skip connection between the cross attention modules is unclear.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xa4I3ph3037", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary\n\nThe authors propose a novel approach to audio-visual action localization.  They introduce a multi-stage cross attention approach to fuse audio-visual features and a consistency loss to enforce temporal continuity.\n\nStrengths\n\nThe approach is well motivated and the technical contribution is clear.  The paper is well written and the contributions are evaluated against a number of recent SOTA approaches on two datasets for action localization.\n\nWeaknesses/Concerns\n\nI would like to see a discussion on the drop in performance in the 3-stage model compared to the 2-stage and 1-stage models in Table 1.  This drop is not intuitive so the authors should provide some analysis here.\n\nIn section 4.3 and Table 2, it seems O-I, O-II and O-III are used before being defined.  Are these results with the 2-stage model?  I would like to see the effect of using the different stage models with the different versions of the loss function.  This will help understand the different contributions to the final performance.\n\nThe model introduces increased computation and additional parameters.  What is the increase in computation?  This could be important for detection systems that prioritize a trade-off of performance and efficiency.\n\nRecommendation and reasoning\n\nThis is a good paper that could be improved with some additional experiments/analysis and improved clarity.  I recommend acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IqLwx3BExyw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a method for predicting the class of an action in a video and localizing the range.\nThe focus is on using both video and audio modalities.\nThey use a cross modal \"multi stage\" attention mechanism, a background classifier, and losses designed to encourage temporal consistency in the output.\n\nReason for Score:\n\nI would accept this paper.\nThe model shows good performance for weakly labelled, multi-model action recognition and localization.\nThe method is well described and would have immediate benefit in practical applications.\n\nPros:\n\nWeakly supervised - avoiding the cost of detailed labelling is very important.\nMulti Modal - Table 1 shows great advantage of using both modalities, and audio is nearly always preset in video. Table 2 shows ablations for the different parts of the loss.\nThe attention mechanism in the model is intuitive. (except the need for multiple stages - see cons).\n\nCons:\n\nI is not clear multi-stage attention is necessary - one stage seems to do well. Did the authors try different hyperparameters (eg. units/dimensions of single attention block) for one-stage of attention. There may be enough room to get the extra 2 percent, and why we should need 2 stages of attention is not clear, what is the intended effect?\n\nThe related Work section is rather plain to read, as the differences with the current work are not explained. It is a long list, and interrupts the flow of the paper without showing how it relates. Also the first sentence in \"Related Work\" does not read well.\n\nSecond paragraph of introduction states that the work intends to \"localize a wider range of actions\". The AVE dataset does not do this - is this a reference to the ActivityNet1.2 dataset - does it contain noisier and \"weaker\" labels than AVE?\n\nSection 3.2 \"snippet-wisely\" is understandable but strange. Also in Fig 2 text: \"respectively without and with L_cont and L_pseu\" is difficult to follow.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "08CZtzA3Fk9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to mix the independent embeddings of audio and visual data by a set of cross-attention layers to the task of audio-visual event localization. They weigh the features based on the correlation between the representations of both modalities to obtain the output representation. After several layers like these, combined with dense skip connections, the final output is a multimodal representation of the input where the features have been individually learned for each modality based on information from the other one. The final features are concatenated and passed through an open-max classifier. Finally, they design different losses to help enforce coherence and continuity among the proposed labels throughout the video.\n\nIn this respect, the only thing that is not clear enough to me is the novelty of the proposed cross-attention layer. While there are some mentions to previous works of multi-modal representation learning, there is no explicit assessment of the contributions of the proposed cross-attention mechanism with respect to previous attempts to build similar representations, or even previous uses of cross-attention for other tasks.\n\nRegarding the experiments, on one hand they perform ablation studies that confirm the contribution of some of the proposed losses and the cross-attention mechanism for building the audio-visual representations, as well as the use of an open-max classifier instead of a soft-max. On the other hand, while an extensive experimental comparison with other recent state-of-the-art models has been performed, proving that the model can improve previous results, it is not clear to me how much of that improvement is due to the use of specific backbones. For instance, the authors use a ResNet152 to extract the visual features, which is a fairly big network. It is unclear to me how much of the improvement over previous state-of-the-art is due to the use of specific backbone networks. In other words, given that even the baseline without cross-attention layers performs close to second, how much is the number of parameters and specific architecture design in charge of building those initial embeddings actually responsible for the final results?\n\nPros of the paper:\nExtensive experimental results.\nImproves state of the art.\nClearly written.\nCons of the paper:\nNovelty of some contributions unclear.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on the task of weakly supervised activity detection (WSAD). The proposed method combines various ideas together: **(i)** a cross-attention module for audio-visual information fusion and better representation, **(ii)** an open-max classifier to treat the background as an open set, and **(iii)** loss terms to encourage temporal continuity of action predictions. The experimental results on well-known benchmark datasets are promising as they beat out many other methods in the literature.

Based on the reviewers' comments, it is clear that the reviewers unanimously see value in the proposed methodology and the competitive results. To strengthen the paper, the authors are encouraged to provide a stronger validation of the contributions being claimed for the specific task being addressed. This would more concretely position the claimed contributions in the WSAD literature. 
Decision: Accept (Poster)