Action Dubber: Timing Audible Actions via Inflectional Flow
Authors: Wenlong Wan
Weiying Zheng
Tianyi Xiang
Guiqing Li
Shengfeng He
Venue: ICML 2025 poster (2025)
Keywords: Video Understanding
Temporal Localization
Primary area: applications->computer_vision
TL;DR: A new method and dataset utilizing inflectional flow to address Audible Action Temporal Localization problem.

Abstract: We introduce the task of Audible Action Temporal Localization, which aims to identify the spatio-temporal coordinates of audible movements. Unlike conventional tasks such as action recognition and temporal action localization, which broadly analyze video content, our task focuses on the distinct kinematic dynamics of audible actions. It is based on the premise that key actions are driven by inflectional movements; for example, collisions that produce sound often involve abrupt changes in motion. To capture this, we propose $TA^{2}Net$, a novel architecture that estimates inflectional flow using the second derivative of motion to determine collision timings without relying on audio input. $TA^{2}Net$ also integrates a self-supervised spatial localization strategy during training, combining contrastive learning with spatial analysis. This dual design improves temporal localization accuracy and simultaneously identifies sound sources within video frames. To support this task, we introduce a new benchmark dataset, $Audible623$, derived from Kinetics and UCF101 by removing non-essential vocalization subsets. Extensive experiments confirm the effectiveness of our approach on $Audible623$ and show strong generalizability to other domains, such as repetitive counting and sound source localization. Code and dataset are available at https://github.com/WenlongWan/Audible623.

Action Dubber: Timing Audible Actions via Inflectional Flow
Wenlong Wan 12
Weiying Zheng *3
Tianyi Xiang 4421
Guiqing Li
Shengfeng He
Abstract
Video without audio
We introduce the task of Audible Action Temporal Localization, which aims to identify the spatio- temporal coordinates of audible movements. Un- like conventional tasks such as action recognition and temporal action localization, which broadly analyze video content, our task focuses on the distinct kinematic dynamics of audible actions. It is based on the premise that key actions are driven by inflectional movements; for example, collisions that produce sound often involve abrupt changes in motion. To capture this, we propose T TA*Net, a novel architecture that estimates in- flectional flow using the second derivative of mo- tion to determine collision timings without rely- ing audio input. TA? Net also integrates self-supervised spatial localization strategy dur- ing training, combining contrastive learning with spatial analysis. This dual design improves tem- poral localization accuracy and simultancously identifies sound sources within video frames. To support this task, we introduce new benchmark dataset, Audible623, derived from Kinetics and UCF101 by removing non-essential vocalization subsets. Extensive experiments confirm the effec- tiveness of our approach on Audible623 and show strong generalizability to other domains, such as repetitive counting and sound souree localiza- tion. Code and dataset are available at https: gi thub. wtthuu..coo/aeee Wen ongwan Audi 6623.
10
TX Not
Audio Source
Audio Position Prediction
Dubbing
Prediction
Video with audio
Figure 1: We propose novel task aimed at temporally lo- calizing audible actions in videos. This task is crucial for automating the dubbing of actions, such jumping and col- lisions, in silent videos, and for facilitating the re-dubbing of sounds in video editing. Our method, TA*Net, em- ploys inflectional flow as foundational prior, establishing promising benchmark for this task.
services like YouTube and TikTok. In particular, movies and television dramas often require dubbing, traditionally labor-intensive process where professionals meticulously identify precise moments for audio insertion. This process is especially demanding for audible actions, such as the sounds of punches in boxing or weapon clashes. We argue that computers can autonomously mark these audible ac- tions, significantly enhancing the efficiency of video clip dubbing, diverging from traditional manual line dubbing, as shown in Fig.
1. Introduction
Videos have become an integral part of daily life on so- cial media platforms, especially on popular video-sharing
"qqual contribution 'School of Computer Science and Engineer- ing. South China University of Technology "School of Comput- ing and Information Systems, Singapore Management University "Scoool of Computing and Data Science, University of Hong Kong "Department of Computer Science, City University of Hong Kong. Correspondence to: Shengfeng He <shhngfenghhe@e @ smu.edu.sg>..
Techniques specifically designed for video clip dubbing ap- plications, particularly for detecting the precise temporal location of audible actions within a video, remain largely un- explored. The closest related fields within computer vision are video action recognition and temporal action localiza- tion. Current methods action recognition (Girdhar et al., 2022: Huang al., 2017; Lin et al., 2019a; Alwassel et al.,
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Action Dubber: Timing Audible Actions via Inflectional Flow
level and an average of 250 frames per video. We have ex- tensively compared our method against existing techniques using our Audible623 dataset, where it demonstrated supe- rior performance. Moreover, leveraging inflectional flow as an early indicator of motion change has proven essential for identifying periodic movements. Our model's effectiveness extends beyond the proposed task, as evidenced by its per- formance in traditional tasks such as repetitive counting in the UCFRep (Zhang et al., 2020) and CountixAV (Zhang et al., 2021) datasets. These results highlight the robustness and versatility of our framework.
Nouth Nution Hand Warring Speakers
Ball Bouncing Hammer Hinting Seccer Inazinng
Figure 2: Instead of invisible audible actions (left), our study targets 'visible audible actions' (right part), where the correlation between visual movements and their asso- ciated sounds enables resolution through computer vision techniques. our research focuses on the computationally tractable challenges in audible actions.
In summary, our main contributions are fourfold:
We present a new task of the audible action tempo- ral localization and establish the first dedicated dataset, Audible623, specifically designed for this purpose.
2020; Feichtenhofer. 2020; Xu et al., 2020b; Bertasius et al.,
2021; Ryali et al., 2023) excel at identifying different types of actions within a video. However, they primarily focus on the semantic aspects of actions rather than their specific kinematic characteristics. This emphasis leads to an over- sight of the internal variations within actions, which are crucial for identifying and differentiating audible moments. On the other hand, video temporal action localization meth- ods (Lin et al., 2018; 2019b; Xu et al., 2020a; Lin et al.,
We tailor an inflectional flow estimation method grounded in the second derivative of the position-time image, aimed at enriching kinematic data with details on object state changes to mimic audible actions.
We propose a novel auxiliary training method featur- ing self-supervvsed spatial localization, which utilizes acquired spatial information boost the network's repre- sentational skills, additionally identifying collision spatial locations as a secondary output.
2021; Xia et al., 2022; Zhang et al., 2022: Cheng & Berta- sius, 2022; Shao al., 2023) are effective at discerning the temporal boundaries between actions, but they often fail to capture the essential spatiotemporal dynamics of specific motion frames that are critical for detecting audible actions. Audible actions are fine-grained moments within the entire action process, making them more challenging to localize compared to coarse- gaarine action boundaries.
We showcase the method's superior ability temporally localize audible actions and its extensive applicability to other tasks, such as conventional repetitive action count- ing and sound source localization.
2. Related Work
In this paper, we present the new task of audible action temporal localization, aimed at predicting the frame-level positions of visible audible actions (see Fig. for clearer def- inition). Central to our approach is the principle that visible audible actions, especially those involving collision sounds, are characterized by inflectional movements, such as abrupt changes during collisions. In response, we design T et, which explores kinematic analysis to develop an inflectional flow estimation method, leveraging the second derivative of displacement detect changes in motion indicative of collisions, thereby enabling the model to identify sound po- sitions without audio input. Additionally, we incorporate self-supervised spatial localization strategy into our training process, enhancing the model's temporal representations through spatial domain analysis. This strategy employs spatial weight information to guide contrastive learning across both inter-video and intra-video contexts, with the side-output indicating the location of audible actions. To overcome the absence of precise labels for audible actions in existing datasets, we created a dataset called Audible623, comprising 623 videos with collision sound actions from Kinetics (Kay et al., 2017). UCF101 (Soomro et al., 2012), and YouTube, with audible actions marked at the frame
Temporal Action Analysis is crucial area in video under- standing, focusing on the recognition, localization, and anal- ysis of actions over time. Temporal action localization aims to identify the categories and localize the timing boundaries of action instances. Methods (Xu et al., 2017: Chao et al., 2018; Lin et al., 2018; 2019b; Xu et al., 2020a; Lin et al., 2021; Xia et al., 2022) first generate potential action propos- als and then use a classifier to predict the action category. Other methods (Linn al., 2017a; Liu & Wang, 2020; Tiru- pattur al., 2021; Zhang et al., 2022; Cheng & Bertasius, 2022; Shao et al., 2023; Huang et al., 2024) achieve local- ization directly through an end-to-end manner. Zhang et al. (Zhang et al., 2022) propose a transformer-baser method to classify moments in videos. Shao et al. (Shao et al., 2023) measure the action sensitivity address the discrepant in- formation of each frame. These methods focus on localizing the entire process of action. In contrast, our approach focuses on identifying the precise timing point of audible actions within the action process. To be specific, the differ- ence between the two tasks is that one focuses on event-level localization, while the other concentrates on accurate key frame identification, Action Counting is another aspect of
2

---

Action Dubber: Timing Audible Actions via Inflectional Flow
temporal action analysis, which quantify the frequency of specific actions in videos. Existing methods (Karvounas et al., 2019; Hu et al., 2022; Dwibedi et al., 2020; Zhang et al., 2020; 2021) estimate action counts by analyzing the periodicity and period lengths of actions. Hu et al. (Hu et al., 2022) suggest applying a transformer to encode multi-scale temporal correlation and further utilize a density map as representation learn action period. Although the periodic- ity of actions is effective for analyzing the action process, it falls short in distinguishing the feaee-level differences within actions.
Examples from our Aud dataset. All videos include at least one visible audible action.
Figure 3:
Table 1: Statistics comparison between our Audible623 and existing temporal action localization datasets (Idrees et al., 2017: Caba Heilbron al., 2015) and repetitive counting dataset (Zhang et al., 2020).
Sound Source Localization (Zhao et al., 2018; Qian et al., 2020; Chen et al., 2021; Fedorishin et al., 2023; Oya et al., 2020; Lin et al., 2023) aims identify the spatial location of the sound source in a video by analyzing the audio signal. Qian et al. (Qian et al., 2020) propose two-stage audio- visual learning framework that performs cross modal feature alignment in rough to fine manner. Chen et al. (Chen et al., 2021) use an automatic background mining technique with differentiable thresholds to incorporate regions with low correlation with a given sound into negative set for com- parative learning. Fedorishin al. (Fedorishin et al., 2023) model the optical flow in videos a prior to better assist in locating sound sources, and utilized cross attention to form stronger audio-visual correlations to achieve visual sound source localization. Here we propose an auxiliary training method that leverages spatial information and produces a source localization map as a side-output.
AudiMe6233
THUMOS14
Activity
UCFRep
# of Videos Actions
623
413
19,994
526
6,262
6,316
23,064
1,276
Average Duration(s)
9.2
4.3
49.2
8,2
Audible Actions
X
X
Key Frame Labeling
X
sitating human annotations. Since we do not use any audio for training and inference, human volunteers are invited to watch each video entirely and inspect the content frame by frame. Initially, volunteers are asked to determine whether the video contains at least one audible action. If video couldn't 't be visually determined for the temporal location of audible frames, it is excluded. Volunteers are then tasked with labeling the frames for each audible action, and the annotations of all audible action frames are converted into key keyframe labels, indicating whether a frame corresponds to audible action.
3. Audible Actions Dataset
Data Collection. As none of the existing video action datasets match the requirement of the proposed audible ac- tion temporal localization task, we need to collect an audible actions video dataset for training and evaluating our method, Initially, we gather videos from YouTube and existing ac- tion video datasets, including Kinetics (Kay et al., 2017) and UCF101 (Soomro et al., 2012), which cover wide range of categories. Despite these datasets contain numerous videos, many do not include audible actions, and some actions lack a clear timing definition. To address this, we filter the videos and select those containing audible actions, such as drum- ming, tennis, and hammer throwing. The collected video should include least one collision event, as we focus on detecting the timing of visual collisions. Several examples from our dataset are illustrated in Fig. 3.
Dataset Statistics. After collecting and annotating the ac- tion videos, we obtain total of 623 videos and allocate 497 videos for training and 126 videos for evaluation. The videos in our dataset have durations ranging from 2.3 to 30.7 seconds, with an average duration of 9.2 seconds. On average, each video consists of 250 frames. Tab. offers a statistical comparison between our dataset and commonly used datasets in temporal action localization. Our dataset furnishes precise keyframe annotations, feature absent in existing datasets yet crucial for the task of audible action temporal localization.
4. Method
Dataset Annotation. To meet the requirement of frame- level detection accuracy, annotations on video frames are crucial. Some datasets, such as THUMOS14 14 (Idrees et al., 2017) and NetivityNet (Caba Heilbron et al., 2015), provide annotation for the start/end times of actions. However, sim- ply considering the midpoint is unsuitable for non-linear actions, and other motion descriptors are unreliable, neces-
Problem Formulation. Given silent video containing audible actions, our goal is to figure out the moment when audible action occurs. Furthermore, we propose a more challenging setting that aims to determine the moment at the frame level, i.c., whether each frame contains action that can generate sound. This contributes to achieving audio-
3

---

Action Dubber: Timing Audible Actions via Inflectional Flow
Kinematicc-boss Inflectional Flow Estimation
Cross kinematic Aggregation Module
Self-supervised Spatial Auxilaary Training
Nan Sepio
Modlan
Motion
Crasp Aegrepied
MFF
MFF
Linear
Soltmax
Smoothness
Contrastive Loss
Temporal Classification Loss
Video Brames
Linsar
Soltmax
Inflectional Floo
Transformer
MPF
Motion Flow Estimation Module
Element-wise Subtraction
Element-wise
Multiplication
Multiplication
Concatenation
Figure 4: Overview of T AA?NNet Our model leverages the novel inflectional flow the supplementary of kinematics prior (e.g., motion) to identify the occurrence of audible actions. We first conduct the kinematics analysis to calculate the proposed inflectional flow one of the kinematics priors under the frand-fro--ooliiionn assumption. Then cross-kinematics aggregation module is introduced to pop-up motion and velocity inflection information from the contexture features of the original image. To further model motion and velocity inflection in the spatial dimension, we propose a self-supervised spatial auxiliary training strategy to localize the motion of the object, both at inter-video level (contrastive loss) and intra-video level (smoothness loss). We then learn classification probabilities O which indicate frames containing audible actions.
visual consistency. Concretely, given video sequence == { "f' I 1, .,2...., with length T, our task is to estimate probabilities for classifying frames into sound frames class and no-sound frames class, represented as O = {O E [0, 1]2 t == 1, 1,2,...,T)
for identifying audible frames. Considering previous ac- tion recognition methods (Lin et al., 2019a; Feichtenhofer, 2020; Ryali et al., 2023) has not fully explored the potential inflection of velocity from motion, we propose to further complete traditional kinetic analysis (i.e., only motion flow) by introducing the novel inflectional flow, motivated by (Xu et al., 2021).
Network Design. The overview of our model is illustrated in Fig. 4. Our fundamental assumption posits that audible ac- tions, specifically collisions, are typically induced by sudden changes in the forces acting upon an object, which manifest as inflections in velocity. Additionally, these collisions are the primary sources of sound. According to this, we delve into the velocity inflection and motion in both temporal and spatial prospective. For temporal modeling, we propose the concept of 'inflectional flow' to signify sudden changes in velocity of an object, enhancing traditional kinematic analy- motion prior. This is complemented by
Inflectional Flow. We consider the video sequence of the object's position over time, that is, 2--positoon-time graph (i.e., x-f graph, where x is the 2D position vector). According to the laws of kinematics, we can get the corre- sponding velocity-time graph (v-t graph) by taking the first order derivative:
dx ut) dt
(1)
sis by serving the integration of backhone encoders and cross-kinematic aggregation module, which together are designed to extract motion-guided contextual features. For spatial modeling, we introduce a self-supervised auxiliary training strategy, constraining the prediction of audible actions through both inter- and intra-video view.
Furthermore, the acceleration-tine graph (a-l graph) is given by the second order derivative:
dxx dt2
a(t)
(2)
According to Newton's First Law, the abrupt application of external force, manifesting acceleration, alters an object's state of motion. This con- cept underpins our analysis, as the acceleration function att) yields critical insights into audible actions. Therefore, we characterize the velocity inflection point, represented by a(t), as the 'inflectional flow', We then integrate the
sudden change in
4.1. Timing Audible Actions with Inflectional Flow
Based on our coand--rom-oollisiin assumption, detected sudden velocity changes can be important indicating factors
4

---

Action Dubber:
Timing Audible Actions via Inflectional Flow
Note that we trim the input sequence at the beginning and the end to ensure that every I has corresponding kinematics features (i.e., my and c²).
Aggregation. To better leverage the guidance from the kinematics prior, we introduce the Cross- Kinematics Aggregation module, which aims to pop-up motion and velocity inflection information from image con- text feature. Specifically, given the tuple consists of image feature and kinematies features (fi, m,; Gi) first project them into (K (()) mat)) ee(s)) respectively using linear layer. Then we calculate the relevance maps as:
Frame f; 1.
Frame
Frame F
Motion flow
Motion Dow
Inilectional flow 01-1
Amf()) (k)) Aer(o) (K()) (5)
Figure 5: Visualization of the motion and inflectional flow. The kinematic analysis gives crucial action prior according to motion flow estimated by optical flow. The light part of a,-1 illustrates the momentary change in the direction of motion.
Vd
vd
where Ams(s) and Acf()) denote the attention ma- trix (Vaswani, 2017) for image-to-motion and image-to-
inflection respectively. The spatial attention is formulated
(i) Viajsoftmax(Anro))
(6)
he(s) = Vyggsoftmax(Ae10
inflectional flow, denoted as a, == a(t), with the motion flow, i2 = vl), serving as novel kinematic prior for our model.
where lm())) and he(s) is the cross-kinematics feature of image-to-motion and image-to-infie coaee-to-inflectionn respectively, and Vi() is obtained by applying linear layer on fi.
In practical, we utilize optical flow to measure vtt) under the assumption that the time interval between two adjacent frames in a video is tiny enough. And we adopt the differen- tial operation on ut) to obtain at)). Given three adjacent frames (4-11 I,, I++1) of a video, we initially compute the forward optical flow (i.e., T;-1 to T,, and T; to 1,-1) as motion flow (v,T,, 1. eitt) using pre-trained optical flow estimation network. Then the inflectional flow is given by:
Finally, we concatenate the three features (fi, lm(t). and he(s) to the cross-kinematics feature, denoted as F..
4.2. Self-supervised Spatial Auxiliary Training
In addition to modeling velocity inflection through temporal dimension, we reconsider in spatial dimension. Here, we adopt self-supervised auxiliary training strategy, consider- ing data labelling for the spatial location of audible action time-consuming and difficult. Specifically, discrimination ability of our model on the motion flow in the spatial dimension by conducting contrastive learning in motion and non-motion areas. On the other hand, we also observe that the motion state changes are continuous, thus a smoothing loss is introduced for constraining.
a v.² -
enhance the
We define the wit, and a as the kinematics prior. Sim- ilarly, we can also obtain the backward kinematics prior, including v, 13-1. and a. The visualization of motion and inflectional flow is showed in Fig. 5.
We introduce three separate encoders (Ex, F.M, and Fc)) with same structure to extract the context feature of the original frames, motion flow, and inflectional flow respec- tively. Each of these three encoders has the same structure, consisting of the first four layers of ResNet50 (He et al., 2016). Specifically, we feed the input tuple in forward direction (4, o/" a²) into the three encoders and obtain corresponding feature (fi, m;; (nnttct) c.). And the motion and velocity inflection features in backward direction (m, and ( can also be calculated given input (wj, ,aT) similarly. We further construct the bidirectional motion feature m, and velocity inflection feature c; by the following concatenation operation:
Spatial Localization of Motion. tion localization methods (Zhang et al., 2022; Shao et al., 2023), they do not consider explicitly learning the spatial motion of the objects. Therefore, we suggest introduc- ing spatial motion localization as an auxiliary training task which aims to reconstruct the motion of object. This strategy can effectively boost the network to distinguish the most dis- criminative area of object motion for better locating frames with potential audible actions.
In previous temporal ac-
Since reconstructing the precise motion of the object is challenging and unnecessary, instead of learning object seg- mentation mask, we relax the constraint by turning to learn a discriminative map, where the motion region is activated with higher probability. In detail, given & cross- cooss-kinematics
m, = Concat(m; ), = (oncat((F.cf (4)
5

---

Action Dubber: Timing Audible Actions via Inflectional Flow
aggregated features {F) j 1.2.....44 k.. the correspond- ing discriminative maps {D, 1, k) through 3 x 3 convolution layer, batch normalization layer, and Sigmoid operation. Here, these & frames are not necessarily from the same video.
ensuring accurate localization. We further propose intra- video temporal regularization loss:
Ltemp. = E,1PE D. 2Di+1ll2.
(12)
where i is the index of a frame from the same video.
Inter-Videw Contrastive Learning. We further conduct the contrastive learning on the motion-area (high-proballe probbbbiiy region) and non-motion area low-probability region) on the discriminative map. We first mask the cross-kinematics ag- gregated feature F'y by both motion region and non-motion region given by Dj separately, formulated as:
The result feed back to the audible action prediction part and train supervised way. Therefore, we can re-detect the action frame by separated action regions.
4.3.
Fusion
Upon obtaining the aggregated features {F, j
FTTh = Pf
Pj
(7)
1.2.....kk) k} and motion region activated features {F) j j= 1,,,,.... ,k], we utilize 3D convolution to encode the temporal and spatial motion information. To extract more detailed information on audible actions, inspired by (Hu et al., 2022), we channel the outputs of 3D convolution into a transformer network equipped with self-attention modules. This setup facilitates the integration of spatiotemporal fea- tures, effectively pinpointing potential instances of audible actions within the video. Then we forward them through fully connected layer to produce the final output O,.
where @ the Hadamard product. F and F denote the motion region activated and non-motion region activated aggregated features respectively. Given F {P p ..2.....k) } and F" = {F" = 1, 1,2..... ,,), the ob- jective of negative contrast (motion region and non-motion region) is given by:
Cne
log(1 1
(8)
p=1
4.4. Objective Function
where (1777 F.T)
F
P I
Furthermore, the positive
For the frame-wise prediction results in the temporal do- main, we conduct supervised training using cross-entropy loss and focal loss (Lin et al., 2017b) as follows:
contrast objective is formulated as:
(F) k:kk 1)
(p=to)pp.. log((Fp: Fg));
Laction = AceLee + A focal:
(13)
(9) where [(ptt]] is 1 for * 9 and otherwise 0, and F can be either from or F". Wp.a represents weights to penalize positive region pairs with less cosine similarity given by ranking, formulated as:
where Are and Afoent are weighting parameters for loss terms, which are set to I and 0.1, respectively. is notewor- thy that we employ soft label technique based on Gaussian augmentation to calculate the Lee-
In summary, the total loss for our TA? is a weighted sum of three losses:
Up. = exp(-a ran)((Fp.F.)),
(10)
Liotal = Aaction Laction + Acont. Ceont. + Atemp, Lremp.s
where a is a hyper-parameter for smoothness controlling and rank((Fp, F&)) represents the rank (Xie et al., 2022) of (Fp: Fg) among all the cosine similarity pair of pm or Fm.
(14)
where Auction, Acont. and Atemp. are weighting parameters for corresponding loss terms, which are set to 1, 0.01 and 0.002, respectively.
The above equations define the objectives for motion to motion region (Lm == C.p((Fm)) and non-motion to non- motion region (Lrr = ())( Finally, the total con-
5. Experiment
trastive loss Loont. is:
Implementation Details. We encode kinematic information using pre-trained ResNet50 (He et al., 2016), mapping to 256 dimensions in the cross cross-kinematies module. During training, we randomly sample 64 frames per video, resizing them to 112 x 112 pixels. The model is trained using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 5e-6, a batch size of 4, and 20k iterations. For inference, we segment the video continuously with step size of 64,
Leant. Line + L.p + LT
(11)
Intra-Video Smoothing Constraint. Notably, the actions in video are typically continuous. Therefore, we refine the localization results to ensure that there are no significant changes in the located regions between consecutive frames,
6

---

Action Dubber: Timing Audible Actions via Inflectional Flow
padding the final segment if it is shorter than 64 frames. For optical flow extraction, we employ GMFlow (Xu al., 2023) due to solid empirical performance and good trade- off between accuracy and efficiency. We implement our method on PyTorch with CUDA, version 1.13. All exper- iments were conducted on a single NVIDIA A800 GPU with 80GB of memory, under Ubuntu 20.04 as the operating system. For model training, we trained the proposed method for 200 epochs, which took approximately 4 hours.
Table 2: Quantitative comparison with associated methods on Audible623 dataset.
Methods
Recallt
Precisiont
Fg
NME]
PME]
BMN
0.417
0,486
0.413
9.327
0.951
ActionFormer
0.323
0.357
0.294
14,663
1.162
TriDet
0.459
0.338
0.328
11.378
1.306
RepNet
0.377
0.428
0.362
10.319
1.100
TransRAC
0,569
0.614
0.553
12.176
0.988
Omnivore
0.410
0.368
0.330
19.454
1.256
TSM
0,314
0.372
0.303
12.160
1.153
Metrics We introduce five metrics to evaluate the per- formance of audible action temporal localization: Recall, Precision, F1, Number Match Error (NME), and Position Match Error (PME). Recall measures the method's ' ability to detect all audible action frames, while Precision evaluates the proportion of accurate predictions among all predicted frames. The Fi score balances precision and recall to pro- vide an overall performance measure. For quantitative and temporal accuracy, NME quantifies the difference between predicted and ground truth audible action frame counts, and PME assesses temporal accuracy, allowing a detection time gap of up to 2 frames from the ground truth. Addition- ally, for counting evaluation, we use mean absolute error (MAE) and off-by-one accuracy (OBO) following the ap- proach in (Dwibedi et al., 2020; Hu et al., 2022). MAE measures the absolute discrepancy between predicted and ground truth counts, normalized by the ground truth, while OBO indicates the proportion of correctly counted instances within one count of the ground truth. Further details are provided in the appendix.
X3D
0.426
0.435
0.392
10.370
1.039
TimeSformer
0,470
0.455
0.405
16.101
1.172
Hiera
0,562
0.436
0.424
20.748
1.265
STG-NF 0.679
0.229
0.342
Ours
0.648
0.656
0.616
3,462
0.744
Table 3: Counting performance UCFRep and CountixAV datasets. Top-2 results are marked in bold and underlined. Our method has not been trained on any counting dataset.
UCFRep
CountixAV MAE]
Method
MAE
OBO
OBO
RepNet
0.915
0.074
0.749
0.231
TransRAC
0.594
0.222
0.686
0.255
X3d
1.245
0.037
0.876
0.192
TimeSformer
0.832
0.0
1.551
0.185
Hiera
0.791
0.152
3.648
0.231
Ours
0.588
0.185
0.549
0.346
Tab. 2, the related methods of temporal action localiza- tion (Lin et al., 2019b; Zhang et al., 2022; Shi et al., 2023) perform bad quality, as they are designed for action recog- nition and localization in long videos. Repetitive action counting methods (Dwibedi et al., 2020; Hu et al., 2022) perform well in classification metries. However, they excel in predicting the information of a single action sequence rather than the instantaneous action of collision. Conse- quently, they can only recognize the approximate position of the action and cannot accurately locate it, lcading to suboptimal performance NME, The metrics for video recognition methods, especially those represented by Hi- era (Ryali et al., 2023) are low. This can be attributed to the limitation of these methods to the utilization of long time sequence action content and their lack of sensitivity to frame-level features. Concerning the video anomaly detec- tion method STG-NF (Hirschorn & Avidan, 2023), while the human keypoint sequence effectively represents actions, it struggles to discern action changes the moment of col- lision. Notably, we do not report NME and PME metrics, as they unfair for snippet level detection. Thanks to the inflectional flow estimation method and the self-superviss spatial auxiliary training strategy proposed in our approach, our method outperforms existing methods in four key met- rics, demonstrating its superiority. Our network effectively utilizes temporal kinematic information in action videos,
5.1. Comparisons with Existing Methods
In this section, we employ several methods for both qual- itative and quantitative comparison with our proposed ap- proach. To the best of our knowledge, there is currently work directly aligning with our setting. Hence, we select eleven methods of four types for comparison based on their relevance to the audible action temporal localization task, including temporal action localization (BMN (Lin et al., 2019b), ActionFormer (Zhang et al., 2022) and TriDet (Shi et al., 2023)), repetitive action counting (RepNet (Dwibedi et al., 2020) and Tra (Hu al., 2022)), video/action recognition (TimeSformer (Girdhar et al., 2022), X3D (Lin et al., 2019a), Omnivore (Feichtenhofer, 2020), TSM (Berta- sius et al.. 2021) and Hiera (Ryali et al., 2023)), and anomaly detection (STG-NF (Hirschorn & Avidan, 2023)). For a fair comparison, we load the corresponding pretrained model (if available) and fine-tuned for the same epochs on Audible623 dataset. All compared methods are trained with the same supervision using our key frame annotations. More details can be found in the appendix.
Quantitative Comparison. We first compare our method with above eleven methods quantitatively. As shown in
7

---

Action Dubber: Timing Audible Actions via Inflectional Flow
Table 4: Ablation results on TA2Nee.
Original Video
Video Motion Infle. Aggr. Geant, Liemp, |Recalff Prec. Fi NME)
PME
0,496
0.395 0.406 5.992 1,021
0.600
0.500 0.501 5.748
0.898
0.609
0.578 0.561 4.420
0.781
0.640
0.597 0.587 3.773 0.806
0.638
0.632 0.601 3.731
0.756
RepNet
0.648
0.656
0.616
3.462 0.744
TRAC
AF
TriDet
tify the approximate range of actions. This limitation arises because their designs focus solely on the duration of actions, resulting in the identification of adjacent frames as audible actions as well. Unlike all the competitors, our method achieves the highest accuracy, which is attributed to the spatial and temporal guidance of kinematics prior obtained through inflectional flow estimation and self- spa- tial auxiliary training, And we have a fine-grained timing prediction for all the audible actions. More comparisons can be found in appendix.
X3D
TimeS.
Hiera
Ours
GT
Figure 6: Qualitative comparison of the visualization of temporal predictions on Audible623 dataset. The first row shows several frames of the input videos, and the second row shows the visualization of temporal probabilities (O,).
5.2. Ablation Studies
Inflectional Flow Estimation Module. To evaluate the impact of the motion and inflectional flow estimation com- ponents, we tested a scenario where prior motion (Motion) and velocity inflection (Infle.) information were excluded, and then progressively integrated this kinematic data. As shown in the first three rows of Tab. 4, the baseline model, which relies only on appearance and temporal information from the video frames, performs the worst among all ablated models. Incorporating motion and velocity inflection data separately improves classification performance by better capturing motion changes. Notably, combining both motion and velocity inflection yields the best results. In contin- uous motion videos, where adjacent frames often change minimally, this component helps focus on come-tt-fframe discrepancies, effectively capturing key actions in audible action sequences.
combining it with spatial motion information to achieve superior results.
Generalization to Counting Task. We demonstrate that the proposed inflectional flow can effectively generalize to repet- itive counting tasks. Without any fine-tuning, we directly compared our method to established counting techniques, including RepNet (Dwibedi et al., 2020) and TransRAC (Hu et al., 2022), on the audible action part of repetitive count- ing datasets. As shown in Tab. 3, our approach set new benchmarks, ranking (Ist, 2nd) on UCFRep and (Ist, 1st) on CountixAV,, despite not being trained for repetitive count- ing. It's important to highlight that RepNet and TransRAC are specifically designed and trained for repetitive counting tasks. The success of our model's ' zero-shot predictions un- derscores the effectiveness of modeling inflectional flows at both temporal and spatial levels. This capability allows for precise identification of keyframes in repetitive actions, such as pinpointing the exact moment of collision in repeated tapping actions, which is crucial for accurate counting.
Cross-kinematics Aggregation Module. We also evaluated the cross-kinematics aggregation module (Aggr.). Here, we directly connected the visual and kinematic features extracted by the encoder to predict sound moments. As shown in the fourth row of Tab. 4, this module enhances performance by focusing on key content features through motion and velocity inflection, effectively capturing audible behaviors in the video.
Qualitative Comparison. We also conduct quali- tative comparison with seven methods, including Rep- Net (Dwibedi et al., 2020), TransRAC (Hu et al., 2022), ActionFormer (Zhang et al.. 2022), TriDet (Shi et al., 2023). TimeSformer (Bertasius al., 2021), X3D (Feichtenhofer, 2020) and Hiera (Ryali et al., 2023). The visualization of the comparative results of temporal localization is shown in Fig. 6. We observe that Hiera fails in the first case, while RepNet mispredicts many actions. ActionFormer and TriDet tend to deer-d audible frames. Although X3D and Tran- SRAC exhibit good prediction accuracy, they can only iden-
Auxiliary Training Strategy with Localization. To as- the effectiveness of our auxiliary training strategy, we analyzed the contributions of inter-video contrastive loss (Loont.) and intra-video temporal regularization loss (Cremp)). ). As shown in the last two rows of Tab. 4, both strategies significantly improve prediction accuracy, partic- ularly precision. Inter-video contrastive learning enhances
8

---

Action Dubber: Timing Audible Actions via Inflectional Flow
Frame 7,-1
Frame
Frame 7++1
Motion flow u/-1
Motion flow eN
Inflectional flow d:-1
Figure 8: Inflectional flow for non-audible actions. By modeling visual motion dynamics, can also captures subtle motion transitions across frames and flow fields, enabling robust temporal analysis.
Figure 7: Visualization of spatial localization results on Audible623 (row 1-2) and AVE (Tian et al., 2018) (row 3- 4) datasets. Our method shows effective spatial localization on the AVE dataset without specific training.
(Audible623) and strong baseline method Net). Our Nel utilizes kinematic analysis for inflectional flow es- timation, identifying potential sound- producing events. We further enhance this with a self-supervised spatial localiza- tion strategy, applying motion analysis both inter- and intra-video levels. Experimental results demonstrate our model's excellence in temporal localization of audible ac- tions and its adaptability other tasks, such repetitive action counting and sound source localization.
the detection of motion state changes by emphasizing spatial positional information, while intra-video constraints help smooth spatial localization, maintaining motion continuity.
5.3. Applications
In this section, we demonstrate the versatility of our method in various applications, including spatial localization and non-audible action analysis.
Limitation. Our approach leverages optical flow to esti- mate the motion flow for further calculating inflectional flow. Therefore, the accuracy of optical flow prediction will affect the prediction quality of the final model. For those challenging video cases like large viewing angle changes, the inaccurate optical estimation will further affect the pre- diction quality of our model. In the future, we will explore more robust motion flow estimation methods.
Motion Spatial Localization. Our proposed auxiliary train- ing strategy effectively discriminates movement areas, shown in Fig. 7. The spatial localization maps, produced side output of our model the Audible623 and AVE datasets (Tian et al., 2018), demonstrate the effectiveness of this strategy in inferring the spatial position of motion within videos containing audible actions. Notably, our method was not trained on the AVE dataset, which highlights the excel- lent generalization ability of the proposed approach.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Beyond Audible Actions. Moreover, we highlight that inflectional flow extends beyond audible actions. As shown in Fig. 8, it performs robustly in non-audible actions as well, demonstrating its versatility across various types of motion analysis. Inflectional flow provides a detailed perspective on state changes within actions, capturing subtle transitions and variations throughout sequences, making it valuable tool for comprehensive temporal action analysis.
A Acknowledgement
This work is supported by the National Natural Sci- ence Foundation of China (Grant 62472180), Guangdong Basic and Applied Basic Research Foundation (Grant 2024A151501 1995))9)), Guangdong Natural Science Funds for Distinguished Young Scholars (Grant 2023B 15 155200097) the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG3-GV-2023. 011), and the Lee Kong Chian Fellowships
6. Conclusion
In this paper, we introduce the task of audible action tempo- ral localization to solve the problem of dubbing silent videos with visible actions and further propose its dedicated dataset

---

Action Dubber: Timing Audible Actions via Inflectional Flow
References
Huang, S., Wang, W., He, S., and Lau, R. W. Egocentric temporal action proposals. IEEE TIP, 27(2)764-7777, 2017.
Alwassel, H., Mahajan, D., Korbar, B., Torresani, L., Ghanem, B., and Tran, D. Self-supervised learning by cross-modal audio-video clustering. NeurIPS, 33:9758- 9770, 2020.
Huang, Z., Xu, X., Xu, C., Zhang, H., Zheng, C., Qin, J., and He, S. Beat-it: beat-synchronized multi-condition 3d dance generation. In ECCV, pp. 273-290. Springer, 2024.
Bertasius, G., Wang, H., and Torresani, L. Is space-time attention all you need for video understanding? In ICML, pp. 813 824, 2021.
Idrees, H., Zamir, A. R., Jiang, Y.-G., Gorban, A., Laptev, Sukthankar, R., and Shah, M. The thumos challenge on action recognition for videos "in the wild". CVIU, 155: 1-23, 2017.
Caba Heilbron, F., Escorcia, V., Ghanem, B., and Car- los Niebles, J. Activitynet: A large-scale video bench- mark for human activity understanding. In CVPR, pp. 961-970, 2015.
Karvounas, G., Oikonomidis, I., and Argyros, A. Reactnet: Temporal localization of repetitive activities in real-world videos. arXiv preprint
Chao, Y.-W., Vijayanarasimhan, S., Seybold, B., Ross, D. A., Deng, J., and Sukthankar, R. Rethinking the faster r-cnn architecture for temporal action localization. In CVPR, pp. 1130-1139, 2018.
2019.
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijaanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et The kineties human action video dataset. arXiv preprint 1rXiv:1705.000 2017.
Chen, H., Xie, W., Afouras, T., Nagrani, A., Vedaldi, A., and Zisserman, A. Localizing visual sounds the hard way. In CVPR, pp. 16867-18876, 2021.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR, pp.
Cheng, F. and Bertasius, G. Tallformer: Temporal action localization with a long-memory transformer. In ECCV, pp. 521. Springer, 2022.
Lin, C., Xu, C.,, Luo, D., Wang, Y., Tai, Y.. Wang. C., Li, J., Huang, F., and Fu, Y. Learning salient boundary feature for anchor-free temporal action localization. In CVPR, pp. 3320-3329, 2021.
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., and Zisserman, A. Counting out time: Class agnostic video repetition counting in the wild. In CVPR, pp. 10387- 10396, 2020.
Lin, J., Gan, C., and Han, S. Tsm: Temporal shift module for efficient video understanding. In ICCV, pp. 7083-7093, 2019a.
Fedorishin, D., Mohan, D. D., Jawade, B., Setlur, S., and Govindaraju, V. Hear the flow: Optical flow-based self- supervised visual sound source localization. In WACV, pp.
Lin, T., Zhao, X., and Shou, 7. Single shot temporal action detection. In ACM MM, pp. 988-998 2017a
2023.
Feichtenhofer, C. X3d: Expanding architectures for efficient video recognition. In CVPR, pp. 203-213, 2020.
Lin, T., Zhao, X., Su, H., Wang, C., and Yang, M. Bsn: Boundary sensitive network for temporal action proposal generation. In ECCV, pp. 3-19, 2018.
Girdhar, R., Singh, M., Ravi, N., van der Maaten, L., Joulin, A., and Misra, Omnivore: A single model for many visual modalities. In CVPR, pp. 16102-16112 12, 2022.
Lin, T. Liu, X., Li, x., Ding, E., and Wen, S. Bmn Boundary-matching network for temporal action proposal generation. In ICCV, pp. 3889-3898. 2019b.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Lin, T.-Y., Goyal, P., Girshick, R. He, K., and Dollár, P. Focal loss for dense object detection. In ICCV, pp. 2980- 2988, 2017b.
Hirschorn, O. and Avidan, S. Normalizing flows for human pose anomaly detection. In ICCV, pp. 13545-13554, 2023.
Lin, Y.B.,, Tseng, H.-Y., Lee, H.-Y., Lin, Y.-Y,, and Yang, M.-H. Unsupervised sound localization via iterative con- trastive learning, CVIU, 227: 103602, 2023.
Hu, H., Dong, S., Zhao, Y., Lian, D., Li, Z., and Gao, S. Transrac: Encoding multi-scale temporal correlation with transformers for repetitive action counting. In CVPR, pp. 19013-19022, 2022.
Liu, Q. and Wang, Z. Progressive boundary refinement net- work for temporal action detection. In AAAI, volume 34, pp. 11612-11619 2020.
10

---

Action Dubber: Timing Audible Actions via Inflectional Flow
Oya, T., Iwase, S., Natsume, R., Itazuri, T., Yamaguchi, S., and Morishima, S. Do need sound for sound source localization? In ACCV, 2020.
Xu, M., Zhao, C., Rojas, D. S., Thabet, A., and Ghanem, В. G-tad: Sub-graph localization for temporal action detection. In CVPR, pp. 10156-10165, 2020a.
Qian, R., Hu, D., Dinkel, H.,, Wu, M., Xu, N., and Lin, W. Multiple sound sources localization from coarse to fine. In ECCV, pp. 292-308. Springer, 2020.
Xu, Y., Han, C., Qin, J., Xu, X., Han, G., and He, Transductive zero-shot action recognition via visually connected graph convolutional networks. 32(8):3761- 3769, 2020b.
Ryali, C.. Hu, Y.-T., Bolya, D., Wei, C., Fan, H., Huang. P.-Y,, Aggarwal, V., Chowdhury, A., Poursaeed, O., Hoff- man, J., et al. Hiera: A hierarchical vision transformer without the bells-and-whistles. In ICML, pp. 29441- 29454, 2023.
Xu, Y., Xu, X., Jiao, Li, K., Xu, C., and He, S. Multi- view face synthesis via progressive face flow. IEEE TIP, 30:6024-6035, 2021.
Zhang, C.-L., Wu, J., and Li, Actionformer: Localizing moments of actions with transformers. In ECCV, pp. 492-510. Springer, 2022.
Shao, J., Wang, X., Quan, R., Zheng, J., Yang, J., and Yang. Y. Action sensitivity learning for temporal action localization. In ICCV, pp. 13457-13469, 2023.
Zhang, H., Xu, X., Han, G., and He, S. Context-aware and -cale-insensitive temporal repetition counting. In CVPR, pp. 670-678, 2020.
Shi, D., Zhong, Y., Cao, Q. Ma, L., Li, J., and Tao, D. Tridet: Temporal action detection with relative boundary modeling. In CVPR, pp. 18857-18866, 2023.
Zhang, H., Li, X., and Bing, L. Video-llama: An instruction- tuned audio-visual language model for video understand- ing. arXiv preprint arXiv:2306.02858, 2023.
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. Xixiv preprint arXiv: 1212.0402, 2012.
Zhang, Y., Shao, L., and Snock, C. G. Repetitive activity counting by sight and sound. In CVPR, pp. 14070-14079, 2021.
Sun, Z., Liang, Y., Ma, Z., Zhang, T., Bao, I.., Li, G., and He, S. Repose: 3d human pose estimation via spatio- temporal depth relational consistency. In ECCV, pp. 309- 325, 2024.
Zhao, H., Gan, C., Rouditchenko, A., Vondrick, C., McDer- mott, J., and Torralba, A. The sound of pixels. In ECCV, pp. 370-586, 2018.
Tian, Y., Shi, J., Li, B., Duan, Z., and Xu, Audio-visual event localization in unconstrained videos. In ECCV, pp. 247-263, 2018.
Zheng, W., Xu, C., Xu, X., Liu, W., and He, S. Ciri: cur- ricular inactivation for residue-aware one-shot video in- painting. In ICCV, pp. 13012-13022, 2023.
Tirupattur, P., Duarte, K., Rawat, Y. S., and Shah, M. Mod- eling multi-label action dependencies for temporal action localization. In CVPR, pp. 1460-1470, 2021.
Vaswani, A. Attention is all you need. NeurlPS, 2017.
Xia, K., Wang, L., Zhou, S., Zheng, N., and Tang, W. Learn- ing to refactor action and co-occurrence features for tem- poral action localization. In CVPR, pp. 13884-13893, 2022.
Xie, J., Xiang, J., Chen, J., Hou, X., Zhao, X., and Shen, L. C2am: Contrastive learning of class-agnostic activa- tion map for weakly supervised object localization and semantic segmentation. In CVPR, pp. 989-998, 2022.
Xu. H., Das, A., and Saenko, K. R-c3d: Region convo- lutional 3d network for temporal activity detection. In ICCV, pp. 5783-5792, 2017.
Xu, H., Zhang, J., Cai, J., Rezatofighi, H., Yu, F., Tao, D., and Geiger, A. Unifying flow, stereo and depth estimation. IEEE TPAMI, 2023.

---

Action Dubber:
Timing Audible Actions via Inflectional Flow
A. More details of dataset
provide the action categories of videos the proposed Audible623 dataset in Fig. 9. should be noted that although Audible623 provides action category annotations, the category information is not used during training or evaluation. The Audible623 dataset comprises categories of audible action videos and 64% of videos have duration over 10 seconds.
We use two datasets, UCFRep (Zhang al., 2020) and CountixAV (Zhang al., 2021), to conduct experiments on repetitive counting tasks. The UCFRep (Zhang et al., 2020) dataset comprises 526 action videos from 23 categories sourced from the UCFI01 (Soomro et al., 2012) dataset, each video is annotated with the intervals of repetitive actions. The CountixAV (Zhang et al., 2021) dataset consists of 1,863 repetitive action videos (some videos have become unavailable) from the Countix (Dwibedi et al., 2020) dataset by filtering out videos with unclear sound or background noise. Then the videos are labeled with the count of repetitive actions. Note that we use these videos with sounds removed as input.
Furthermore, we apply AVE (Tian al., 2018) dataset for the visualization of spatial localization results. AVE (Tian et al., 2018) comprises 4,143 videos of visual events and the corresponding audio, covering 28 action categories. Each category includes at least 60 videos.
Tapping
Bouncing
Battlerope
Pingpong
Clapping
GolfSwing
5.1%
6.3%
Boxing
4.2%
10.6%
3.4%
Drumming
CricketShot
Slicing Others
Tennis
28.1%
16.5%
Soccerjuggling
Hammering
Figure The distribution of videos in different categories of Audible623 dataset.
B. Metrics
We utilize five metrics to evaluate our proposed Audible Action Temporal Localization task, including Recall, Precision, Fi Number Match Error (NME), and Position Match Error (PME). These metrics are formulated as follows:
NME
C
C
GT
(15)
N N
1
m:
PME
GT
(16)
where N and m, the number of vidcos and matched audible action frames, respectively. c; ΣT O and GT ΣT, OCT indicate predictions and ground truths (GTs) of the sum of audible actions, respectively. I, and 18T are predictions and GTs of the temporal location audible action frames, respectively.
12

---

Action Dubber: Timing Audible Actions via Inflectional Flow
Table 5: Ablation study over the weight of Lcont. and Cremp.
Acant.
Atemp.
Recallt
Prec.1
Fi f
NMEJ
PME|
0.640
0.597
0.587
3.773
0.806
0.1
0.628
0.629
0.591
3.655
0.762
0.01
0.638
0,632
0.601
3.731
0.756
0.001
0.639
0.614
0.592
4.168
0.756
0.01
0.02
0.636
0.628
0.596
3,932
0.766
0.01
0.0002
0.671
0.620
0.607
3,924
0.792
0
0.002
0,625
0,627
0.587
4.252
0.772
0.1
0.002
0.638
0.617
0.597
3,487
0.754
0.001
0.002
0.667
0.611
0.603
4.622
0.820
0.002
0.648
0.656
0.616
3.462
0.744
Following the previous repetitive counting work (Dwibedi et al., 2020; Hu et al., 2022), the Oft-By-One Error (OBO) and Mean Absolute Error (MAE) metrics for action counting can be also formulated as:
N
1 N
OBO =
c
GT
1
(17)
MAE
(18)
1=1
C. Comparison Methods
The relevant details for all comparison methods are as follows:
Temporal Action Localization. We choose three temporal action localization methods, including BMN (Lin et al., 2019b), RorionnForen al., 2022) and TriDet (Shi al., 2023). Specifically, we employ them frame-level features input and predict per frame performance. Specifically, for BMN, we use two-stream network that incorporates both video and optical flow features input.
Repetitive Action Counting. We select two state-of-the-art repetitive counting methods for comparison, including RepNet (Dwibedi et al., 2020) and TransRAC (Hu et al., 2022). These methods predict the number of repeated actions in the video by aggregating the probabilities of the final output. We adjust the final classifier to ensure it outputs frames with probabilities greater than a predefined threshold.
Video/Action Recognition. We adopt five state-of-the-arr video/action recognition methods, including TimeSformer (Berta- sius al., 2021), X3D (Feichtenhofer, 2020), Omnivore (Girdhar et al., 2022), TSM (Lin et al., 2019a) and Hiera (Ryali al., 2023), which recognize the categories of actions present in the video, such as shaking hands, hugging, etc. Similarly, we keep the original feature extraction module unchanged and only adapt the final classifier to make frame-level predictions.
Anomaly Detection. We choose STG-NF (Hirschorn & Avidan, 2023), a video anomaly detection method that solely utilizes human pose (Sun et al., 2024) to learn spatial and temporal relationships. Specifically, we extract human pose keypoints for videos the Audible623 dataset and modify the sliding window to detect the presence of audible actions within the window.
D. More Ablation Study
We further report the results of the ablation study over the weight of inter-video contrastive loss (Coont.) and intra-video temporal regularization loss (Ezemp.) in Tab. 5. The results demonstrate that adding Coont. and Ltemp. can improve the performance of our method. However, using only Ctemp. resulted in increased recall but decreased precision, suggesting that solely maintaining spatial attention unchanged might lead inaccurate localization.
13

---

Action Dubber: Timing Audible Actions via Inflectional Flow
Frame 7--1
Frame I,
Frame I,+1
Motion flow 04-1
Motion flow 24
Inflectional flow a,-1
GT
Ours
Figure 10: Visualization of failure case.
E. More Qualitative Comparison
We also present more qualitative comparisons, visualizing temporal predictions on the proposed Audible623 dataset. We provide video frames and temporal prediction results, conducting a comparison between our method and several existing approaches (Zhang et al., 2022; Shi et al., 2023; Dwibedi et al., 2020; Hu er al., 2022; Feichtenhofer, 2020; Bertasius et al., 2021; Ryali et al., 2023; Zheng et al., 2023). These comparisons are illustrated in Fig. 12 and Fig. 13. It evident from the visualizations that our method outperforms these approaches.
F. Failure Case
As our method is based on the kinematic analysis for inflectional flow estimation, it relies on the accuracy of inflectional flow. We illustrate a failure instance in Fig. 10. The perceptible decline in the quality of optical flow estimation stems from camera viewpoint adjustments. Consequently, this degradation significantly impacts our kinematic analysis, resulting in our method's failure detect audible actions frame I; (highlighted by red circle).
G. Evaluation using Vision Language Model
We also test the performance of large-scale video understanding model (Video-LLLaMA (Zhang et al., 2023)) on the task of audible action temporal localization. As shown in Fig. 11, despite the special requirements (localizing the exact time frame of audible actions) made the prompt, the results indicate that while these large models can broadly describe the actions occurring video, they lack the capability to accurately pinpoint the timing of actions, especially the precise moments of audible action.
14

---

Action Dubber: Timing Audible Actions via Inflectional Flow
Based the video content provided, please analyze the video to pinpoint the point in time of each kicking action, output the exact frame number where the ball was kicked in each kicking action, and then compile this information in the order it occurred.
The following compilation of the exact frame numbers where the ball kicked in each kicking action in the provided video, in the order it occurred:
Video Content: A young boy is shown playing soccer green field. He is wearing white shirt and shorts. He kicking the ball and running around the field.
Kicking Action1: Frame 20-22 Kicking Action2: Frame 30-32 Kicking Action 3: Frame 40-42 Kicking Action4: Frame 50-52 Kicking Action5: Frame 60-62 Kicking Action 6: Frame 70-72
O
Actually, the exact frame numbers the video where the ball is kicked in each action 11, 44, 61, 79.
Figure 11: Applying Video-LLaMA (Zhang et al., 2023) to Temporal Audible Action Localization Task. We found failed accurately localize the exact time frame of audible actions.
H. Demo
present the visualization audible action temporal predictions and spatial localization in dynamic form in supplementary video. Additionally, we demonstrate some results on video re-dubbing key application fully exploit model.
15

---

Action Dubber: Timing Audible Actions via Inflectional Flow
GT
ActionT ormer
TriDet
RepNet
TrenRAC
TimeSooree
Hiera
Ours
GT
ActionFormer
TriDet
RepNet
TransRAC
X3D
TimeSformer
Hicra
Ours
AetionFFormer
TriDet
RepNet
TResRAC
X3D
TimeSformer
Hiera
Ours
GT
ActionFormer
TriDet
RepNet
TrasRAAC
X3D
TimeSforner
Hiem
Ours
Figure 12: Visualization of temporal predictions the Audible623 dataset.
16

---

Action Dubber: Timing Audible Actions via Inflectional Flow
ActionFormer
TriDet
RepNet
TransRAC
TimeS former
Hiera
ActionFormee
TriDet
RopNet
TransRAC
X3D
TimeSfommer
Hiera
Ours
AstionFormer
TriDet
RepNet
TransRAC:
X3D
TimeSformer
Thera
Ours
GT
ActionForrmer
TrDet
RepNet
TransRAC
X3D
TfoeSorreer
Hiera
Ours
Figure 13: Visualization of temporal predictions on the Audible623 dataset.
17


Reviews:
[{"review_id": "yZ8jvXKSKt", "reviewer": "Reviewer_v1Hi", "summary": "This paper introduces the task of audible action temporal localization and proposes a novel framework $TA^2Net$ alongside the Audible623 dataset. The method features a inflectional flow estimation technique grounded in the second derivative of the position-time images. Additionally, the authors develop a self-supervised spatial localization. The effectiveness of the proposed method is demonstrated, and its broad applicability is validated in other tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. User Study**\n\nAs suggested, we conducted a user study comparing five approaches: TriDet, TransRAC, X3D, Hiera, and our proposed method. We curated a set of eight videos and enlisted 30 participants for the study. Each video was dubbed using sound aligned to the audible action locations detected by each method. Participants were then asked to assess the audio-visual synchronization quality and select the top two methods they perceived as most accurate for each video. The results show that our method was most frequently selected as the best, indicating a clear preference and superior perceived synchronization performance.\n\n| Method   | Top-2 Rate|\n|----------|----------|\n| Ours     | 0.8458 |\n| TriDet   | 0.2208 |\n| TransRAC | 0.5625 |\n| X3D      | 0.1125 |\n| Hiera    | 0.2583 |\n\n**2. Concerns on Short Duration & Application Scenarios**\n\nWe respectfully disagree with this concern, as it appears to stem from a misunderstanding of both the dataset characteristics and our method’s design. The video duration and frame rate in the Audible623 dataset, as well as in other benchmarks used in our study, are consistent with established standards in the action counting and localization community. These datasets are widely adopted to ensure fair comparisons and reproducibility. More importantly, our method operates at the frame level, making it inherently agnostic to the overall video length. This design choice allows the method to scale effectively to longer and higher-framerate videos without degradation in localization performance. As such, our approach remains robust and applicable to real-world scenarios, including platforms like YouTube and professional media content.\n\n**3. Methodological Ambiguity in Motion Feature Extraction**\n\nWe respectfully believe this concern may stem from a misunderstanding of motion feature roles. Depth and disparity capture static spatial structure but do not reflect temporal dynamics. For example, counting the number of times a person jumps depends on detecting repeated vertical motion—something depth alone cannot convey. In contrast, optical flow reflects dynamic changes across frames and is widely recognized as a reliable signal for temporal modeling.\n\nWhile depth can theoretically be used to extend 2D optical flow to 3D, such usage is rare in practice, as action scenarios that truly benefit from 3D motion modeling are uncommon and often domain-specific. Moreover, incorporating depth adds computational overhead without clear gains for typical action tasks. \n\nWe chose Xu et al.’s network for its solid empirical performance and a good trade-off between accuracy and efficiency. While alternatives like FlowNet or PWC-Net are well-known, we did not observe significant advantages for our task. We will clarify this rationale in the revised manuscript.\n\n\n**4. Optical Flow Method Selection**\n\nCompared to traditional methods such as DisFlow, modern neural network-based approaches offer significantly improved robustness in challenging real-world scenarios involving occlusion, noise, or complex motion, and they perform consistently well across varying resolutions. Among these, GMFlow demonstrates state-of-the-art accuracy while maintaining inference times comparable to established methods like PWC-Net [1, 2]. Additionally, GMFlow offers a more streamlined and efficient framework for computing bidirectional optical flow, which is beneficial for our application. Based on these advantages and after thorough consideration, we selected GMFlow for optical flow extraction in our pipeline.\n[1] Xu H, Zhang J, Cai J, et al. Gmflow: Learning optical flow via global matching, CVPR, 2022.\n[2] Teed Z, Deng J. Raft: Recurrent all-pairs field transforms for optical flow, ECCV, 2020.\n\n**Note to Reviewer**\n\nWe appreciate the reviewer’s time and effort in evaluating our work. While we find that some of the comments may stem from misunderstandings of our method and its design choices, we have addressed each point in detail and clarified the rationale behind our decisions. We believe these clarifications resolve the concerns raised and further highlight the strengths and contributions of our approach. We respectfully ask the reviewer to reconsider the insights and potential impact of our work in light of these responses.", "claims_and_evidence": "please refer to Weaknesses.", "methods_and_evaluation_criteria": "While the proposed dataset demonstrates partial validity for audible action temporal localization, its ability to capture real-world complexity remains questionable. The current evaluation primarily focuses on short-duration, scripted scenarios (e.g., 9.2-second clips in Audible623) with controlled audio-visual correspondence.", "theoretical_claims": "I have reviewed the theoretical claims in Sections 4.1 (Timing Audible Actions with Inflectional Flow) and 4.2 (Self-supervised Spatial Auxiliary Training). While the presented formulations are logically consistent and free of apparent errors, their relative simplicity warrants discussion. The derivations primarily involve elementary operations without addressing more complex scenarios (e.g., non-linear motion patterns).", "experimental_designs_or_analyses": "I did. The experimental design and analysis are sound and valid. The authors have provided.", "relation_to_broader_scientific_literature": "please refer to Weaknesses and Questions For Authors.", "essential_references_not_discussed": "please refer to Weaknesses and Questions For Authors.", "other_strengths_and_weaknesses": "### Strengths\n\n1. **Novel Task FormulationL:** Proposes the first formal task of audible action temporal localization.\n2. **Rigorous Validation：** Demonstrates broader applicability through:\n    * Cross-task generalization experiments\n    * Vision-language model (VLM) evaluations in supplements\n\n### Weaknesses\n\n1. **Lack of user study:** A potential issue with $TA^2Net$ is that it shows slightly weaker performance on certain metrics compared to some baselines for the audible action temporal localization task, which may make the subjective human opinions of higher importance to the present manuscript. This should further validate how well the paper's core assumption aligns with the actual human preferences.\n2. **Mismatch Between Data Characteristics and Claimed Application Scenarios:** The proposed method is primarily evaluated on short and low-framerate videos (e.g., Audible623 dataset, with an average duration of 9.2 seconds and 250 frames per video). While this setup may suffice for initial validation, it significantly limits the generalizability of the method to real-world applications. In practice, videos on platforms like YouTube, TikTok, or in professional contexts (e.g., movies, TV dramas) are typically longer and have higher framerates, requiring more robust temporal modeling and scalability. The current experiments do not adequately address these challenges.\n3. **Methodological Ambiguity in Motion Feature Extraction:** The paper’s reliance on Xu et al.'s pre-trained optical flow network raises concerns about feature selection rationale. While the chosen network can extract multi-modal motion features (e.g., depth, disparity), the authors exclusively utilize optical flow without justifying why supplementary features were disregarded. This omission is particularly notable given that depth/disparity information could enhance temporal modeling in complex scenes. Furthermore, the decision to adopt Xu et al.'s architecture over established alternatives like FlowNet or PWC-Net lacks explicit justification. Critical factors such as computational efficiency (e.g., inference speed comparisons), benchmark performance, or inherent architectural advantages for audible action temporal localization tasks remain unaddressed. These unresolved design choices cast doubt on whether the selected framework optimally balances accuracy and practicality. To strengthen the methodology, the authors should either provide empirical evidence or theoretical arguments explaining why optical flow alone suffices for their task.", "other_comments_or_suggestions": "please refer to Weaknesses.", "questions_for_authors": "The paper computes forward optical flow between consecutive frames using a pre-trained network (Xu et al., 2023). Could the authors clarify why this approach was prioritized over classical optical flow algorithms (e.g., DisFlow, Brox Flow) that are widely adopted in video processing?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5YbXQRTlmq", "reviewer": "Reviewer_umcX", "summary": "This paper introduces a new task called audible action temporal localization, aimed at predicting the frame-level positions of visible audible actions.\n\nAnd the paper further proposes a dedicated dataset called Audible623, derived from Kinetics and UCF101.\n\nFinally, the paper proposes a baseline method TA$^2$Net which employs flow estimation based on motion's second derivative.\n\n## update after rebuttal\nI keep my initial rating of weak accept. The rebuttal addressed some of my concerns.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Ambiguity on Timing Boundaries of Actions**\n\nTemporal boundary ambiguity is indeed a well-known and widely acknowledged challenge in general action understanding tasks. Precisely defining the start and end of an action, particularly for semantically complex or continuous activities such as \"sit down\" or \"throw the ball\", is inherently difficult due to visual similarity across adjacent frames and the subjective nature of temporal segmentation.\n\nHowever, unlike general action semantic understanding, our task, Audible Action Temporal Localization, is significantly more constrained and objective. Rather than requiring full temporal segmentation of action intervals, we focus on identifying the specific frames in which a sound is produced by an action. These \"sounding frames\" are typically easier to annotate consistently, as the accompanying audio provides a clear and reliable temporal anchor.\nWe will clarify this annotation protocol and emphasize the inherent simplicity and objectivity of our task in the revised Dataset Annotation section.\n\n**2. Concerns on Dataset Categories**\n\nOur task is fundamentally different from the original objectives of Kinetics and UCF101. While those datasets are designed for action classification across a broad range of semantic categories, our focus is category-agnostic, targeting the detection of action patterns rather than semantic understanding. As a result, the Audible623 dataset does not include category annotations but instead provides precise temporal labels for frames where audible events occur. This design is well aligned with the goals of dubbing and audio-visual synchronization, which rely on accurate temporal localization rather than action recognition.\n\nThanks to the design of our framework, our method demonstrates strong generalization ability. As shown in Table 3, it performs well on datasets with different action categories, such as UCFRep and CountixAV. This confirms that our approach is not constrained by specific categories and can generalize effectively to unseen actions. For instance, our method was successfully applied to animated content in the demo without any additional training.\n\nWe will clarify these distinctions and elaborate on the rationale behind our dataset design in the revised manuscript.\n\n**3. Audible Action Definition**\n\nWhile audible actions are a subset of general actions, they serve a distinct and crucial role in applications like sound dubbing and repetitive counting. Unlike general action recognition, which often relies on high-level and potentially ambiguous semantic categories (e.g., \"playing sports,\" \"interacting\"), audible actions are grounded in concrete, low-level visual cues that correlate directly with sound production (e.g., clapping, stomping, tapping). This distinction makes it category-agnostic and more robust to domain shifts. This targeted focus enables reliable grounding of actions with sound, which is essential for many practical tasks.\n\n**4. Reuse of TAL Datasets**\n\nAs mentioned in Section 3.2, existing Temporal Action Localization (TAL) datasets are not directly applicable to our task. The keyframe-level labels required for Audible Action Temporal Localization cannot be reliably derived from the start and end frames of action segments in these datasets. In many cases, the annotations are relatively coarse and do not separate repeated instances of the same action. For example, a kicking sequence with multiple discrete kicks is often labeled as a single continuous segment, whereas our task focuses on detecting each individual audible event.\n\nAdditionally, TAL datasets include many inaudible actions that fall outside the scope of our task. Reusing these datasets would require significant re-annotation and filtering to align with our objective. For these reasons, we chose to construct a dedicated benchmark with precise frame-level annotations that better support the goals of audible action localization and audio-visual synchronization.", "claims_and_evidence": "1.\tThe authors introduce the task of Audible Action Temporal Localization, aimed at pinpointing the spatio-temporal coordinates of audible movements. However, there are no spatial annotations in the proposed dataset. The authors propose an auxiliary training method that leverages spatial information and produces a localization map as a side-output.\n\n2.\tThe authors claim that the difference between Temporal Action Analysis and Audible Action Temporal Localization is that one focuses on event-level localization, while the other focuses on accurate key frame identification. And the authors formulate the task as aiming to determine whether each frame contains action that can generate sound. However, the timing boundaries of certain action in the video inevitably exhibit some degree of ambiguity. The authors have not discussed this ambiguity issue in the Dataset Annotation section. \n\n3.\tThe proposed dataset is derived from Kinetics and UCF101 which contains at least 400 and 101 action classes respectively. And the proposed dataset contains only 14 categories of audible actions. The reviewer has concerns about the generalizability of the proposed dataset and its applicability in helping the dubbing task.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n\n1. Proposing a new task, constructing a dedicated dataset and proposing a baseline method with comprehensive comparisons requires a significant amount of effort. And the paper is written in a relatively clear and readable manner.\n\nWeaknesses\n\n1. The definition of visible audible action remains ambiguous. And judging from the proposed dataset, the audible action is merely a small subset of the existing action categories.", "other_comments_or_suggestions": "1.\tTypos in the caption of Figure 4 (Page4-Line181).", "questions_for_authors": "Since the proposed new task is close to the task of traditional Spatial Temporal Action Localization/Temporal Action Detection, why would the authors construct the new benchmark using the public datasets on these tasks? It seems the temporal annotation would be easier and the spatial-temporal annotations can be reused.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P0dFNVlUkr", "reviewer": "Reviewer_kBge", "summary": "The paper proposes a new task named \"audible actions\" and introduces a new dataset for this task. It introduces an inflectional flow estimation and an auxiliary self-supervised training method to improve the performance. Experiments are conducted on Audible623, UCFRep, and CountixAV datasets.\n\n## update after rebuttal\nI keep my initial rating of weak accept after the rebuttal. The rebuttal addressed most of my concerns.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Distinction Between Our Method and TAL Methods, and Applicability to TAL**\n\nWe respectfully clarify that our method targets a task that is fundamentally different from traditional TAL. While TAL focuses on localizing the full temporal extent of actions, our method is designed to identify precise keyframes where audible events occur. In this sense, TAL addresses event-level localization, whereas our approach focuses on frame-level detection tied to sound-producing moments.\n\nThis distinction is important, as audible frames often represent brief, high-impact instants that are not captured by coarse action intervals. As shown in the appendix, our method also performs well on non-audible actions, demonstrating its ability to capture subtle temporal transitions. We believe this fine-grained capability may provide useful insights for improving temporal precision in future TAL work.\n\n**2. Generalization Beyond the Training Dataset**\n\nPlease refer to the reply of Reviewer umcX Q2.\n\n**3. Ambiguity of Audible Action && Multiple Audio Sources**\n\nWhile the exact decibel level may vary, our task focuses on visually observable actions that are intentionally associated with sound production, consistent with how humans infer sound-related events from visual cues.\nTo minimize subjectivity, we annotate dominant and intentional audible actions during training, following standard practice in broader action recognition tasks where annotators focus on salient or representative instances of an action (e.g., annotating the main swing in \"golf swing\" rather than every subtle motion). \n\nFor testing, our model responds to motion patterns indicative of sound, regardless of their intensity. For instance, in the case of basketball dribbling, both soft and forceful dribbles share similar motion dynamics, and our model is designed to detect both.\n\nWe also clarify that Fig. 1 is purely for illustration purposes; all samples used in training and evaluation come from real-world, in-the-wild videos with natural variations in appearance and motion.\n\nWe will revise the manuscript to better explain our annotation criteria and the illustrative role of Fig. 1.\n\n**4. Multiple Audio Sources**\n\nIn scenarios involving multiple audio sources, our method is designed to detect all sound-producing actions present in the video. However, distinguishing between individual sound sources is beyond the current scope of our work. Our primary objective is to accurately identify the timing of audible actions, regardless of their specific source.\n\n**5. Quantitative Comparisons with TAL methods**\n\nWe further evaluated BMN on the Audible623 dataset using a two-stream network that incorporates both video and optical flow features. This allows for a fair comparison with TAL methods that also leverage motion information. Since the source code for Zhang et al. and Liu et al. is not publicly available, we have contacted the authors and will include further discussion in the revised manuscript based on their responses.\n\n| Methods | Recall↑ | Precision↑ |  F1↑  |  NME↓  |  PME↓ |      \n| :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |  \n|  BMN    | 0.417  |   0.486   | 0.413 | 9.327 | 0.951 |     \n|  Ours   | 0.648  |   0.656   | 0.616 | 3.462 | 0.744 |\n\n**6. Concerns on Computational Complexity**\n\nThanks to our lightweight network design, the computational cost of incorporating additional optical flow and inflectional flow features remains manageable. We evaluated the processing time of various methods on the same set of videos. While the inclusion of optical flow does introduce some additional overhead, our method maintains a favorable trade-off and still outperforms most competing approaches in both accuracy and efficiency.\n\n| Methods                        | Time(s) | Methods                     | Time (s)  |\n|--------------------------------|---------|-----------------------------|---------|\n| RepNet                         | 0.075   | Hiera                       | 0.263   |                                           \n| TransRAC                       | 4.301   | ActionFormer                | 0.233   |\n| X3D                            | 0.066   | TriDet                      | 0.240   |\n| TimeSformer                    | 9.968   | Ours (w/ & w/o Flow)    | 0.152 / 0.083 |\n\n**7. Dataset Selection and Annotation**\n\nAnnotators were instructed to review each video individually and select visually observable audible actions, excluding videos that either lacked audible actions or featured occluded actions. For high-frequency actions such as drumming, which are particularly challenging to annotate precisely, we adopt a frame-level labeling strategy, marking only the frames where the drumstick makes contact with the drum.\n\nAs noted in Sec. 3.2, we do not apply category-based filtering during annotation. Instead, the first round of filtering involves manually reviewing each video to determine whether it contains visually identifiable audible actions.", "claims_and_evidence": "The paper propose \"audible actions\" as a novel task in TAL. However, it is a subset of the existing TAL task where the actions have audible cues and so existing methods can still work. So, the claim of novelty is not justified.\n\nThe proposed method is trained to identify \"audible actions\". But it can only identify actions that it was trained on. Does it generalize to audible actions beyond the training dataset? There is no discussion on this.\n\nFigure 1 shows a motivating example of audible actions in computer video games. However, visible audible actions are subjective and ambiguous in video games making it difficult to quantify. Further, certain actions such as dribbling a ball can be considered audible or not depending on the decibel level. In essence, the framing of this task is subjective and there is no clear rubric for it.\n\nIn real videos, audible actions can occur simultaneously which can result in commotion if there are multiple sources. For example, two persons juggling a ball. How to distinguish between different audio sources in this case?", "methods_and_evaluation_criteria": "The self-supervised spatial auxiliary training (Sec. 4.2) loss is not very well motivated and it appears to be added just to improve the overall performance.", "theoretical_claims": "No theoretical claims made in the paper.", "experimental_designs_or_analyses": "Missing quantitative comparisons with popular TAL methods such as BMN and G-TAD.", "relation_to_broader_scientific_literature": "The key contributions of the paper in introducing the task of predicting audible actions and proposing the model TA2Net. Since, this is a narrow  subset of tasks defined in existing TAL methods, the contribution is limited to a narrow domain that does not benefit the broader TAL community. For example, it would be better if the proposed method can still apply to broader TAL tasks.", "essential_references_not_discussed": "The paper does not cite prior methods which have used optical flow for TAL tasks such as \n\nDejun Zhang, Linchao He, Zhigang Tu, Shifu Zhang, Fei Han, Boxiong Yang, Learning motion representation for real-time spatio-temporal action localization, Pattern Recognition, Volume 103, 2020.\n\nYuanzhong Liu, Zhigang Tu, Liyu Lin, Xing Xie, and Qianqing Qin  Real-time Spatio-temporal Action Localization via Learning Motion Representation, ACCV, 2020.", "other_strengths_and_weaknesses": "The paper does not discuss the computational complexity of the method, specifically because it uses optical flow  which is expensive.", "other_comments_or_suggestions": "Typo - should be \"quantitatively\" in L376", "questions_for_authors": "What are guidelines for annotators in choosing audible actions?\n\nHow is data annotated for high frequency actions like drumming? Are all frames labeled as audible?\n\nFor dataset collection, how does the filtering occur? With action class names? (L152)\n\nHow to deal with categories that are not labeled but present in the video? i.e., How does it generalize to novel audible actions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VCCx6r9xXZ", "reviewer": "Reviewer_WLLu", "summary": "Authors design a novel task of audible action temporal localization. A new dataset called $Audible623$ and a $TA^2Net$ architecture are proposed for this task. Experiments shows the advantages of them.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Formatting Issues**\n\nWe sincerely appreciate your valuable feedback. We will revise the figure citations, adjust the formatting of Figure 2, correct the subscripts in the equations, and improve the overall layout of the paper. Additionally, we will fix the typographical error in line 376 to ensure clarity and consistency throughout the manuscript.\n\n**2. Concerns on Dataset Selection**\n\nWe selected Kinetics and UCF101 because they serve as standard benchmarks for action counting, which is a core use case of our task. These datasets include a wide range of repetitive, sound-associated actions such as “clapping”, “jumping jacks”, “typing”, and “punching”, which are well aligned with the objectives of audible action localization.\n\nIn contrast, datasets like SSv2 and HMDB51 emphasize abstract or context-dependent interactions that are less relevant to our goal. For example, SSv2 includes many object-centric and non-audible actions such as “pretending to open something” or “moving something from left to right”, which lack consistent visual-sound correspondence. Similarly, HMDB51 features a high proportion of semantically complex or non-repetitive actions such as “smiling” or “listening to music”, which are not suitable for sound grounding or counting tasks.\n\nTherefore, our dataset selection is driven by the need to support visually grounded audible action detection, and we believe it is well justified for the specific focus of our work.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The correctness of any proofs for theoretical claims is suitable.", "experimental_designs_or_analyses": "The experiment design is suitable.", "relation_to_broader_scientific_literature": "The task authors proposed is diverges from traditional action recognition and temporal action localization.", "essential_references_not_discussed": "Related works are introduced.", "other_strengths_and_weaknesses": "**Strengths:**\n1. The proposed task of audible action temporal localization is interesting and the motivation is reasonable.\n2. The explanation of the task in Figure 1 is clear and vivid.\n3. Clear model structure.\n4. Reviewer appreciate the experimental results in demo videos.\n\n**Weaknesses:**\n1. The cite of Figure 2 precedes Figure 1 in the introduction which is strange. Additionally, the figure name is main text is Figure X instead of Fig.X. Reviewer recommend authors check and apply \\figurename~\\ref in their manuscript.\n2.  Why Figure 2 is not PDF format and fonts within it cannot be selected? Authors are suggested to polish Figure 2.\n3. Why authors select Kinetics and UCF101 instead of other widely-used video datasets such as SSv2 and HMDB51?\n4. The formula is not properly written, especially the subscripts of the loss. In standard mathematical formulas, the subscripts of such nonvariables should not be used in italics but in regular fonts.\n5. The layout of the manuscript should be improved. There are many unreasonable single words that take up an entire line.", "other_comments_or_suggestions": "Please refer to weaknesses.", "questions_for_authors": "Please refer to weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a new task of audible action temporal localization and a new dataset for this task, namely Audible623. Different from traditional Temporal Action Localization (TAL) task, where the model is tasked to localize (temporally) instances of a set of pre-defined actions. Audible TAL is formulated as detecting moments (keyframes) from the videos that make sound. The paper also presents a novel approach with solid experiments.

After rebuttal, most of concerns raised by reviewers are addressed by the authors. And all reviewers recommend weak accept. AC reads all reviews, rebuttal and discussions and agrees with the reviewers, thus recommends a weak accept.

A few minor comments:
  * The dataset is quite small ~ 600 videos which may make the findings less convincing.
  * Missing a few references: [1] Zhao et al., The Sound of Pixels, ECCV 2018, [2] Alwassel et al., Self-Supervised Learning by Cross-Modal Audio-Video Clustering, NeurIPS 2020.
Decision: Accept (poster)