$F^3Set$: Towards Analyzing Fast, Frequent, and Fine-grained Events from Videos
Authors: Zhaoyu Liu
Kan Jiang
Murong Ma
Zhe Hou
Yun Lin
Jin Song Dong
Venue: ICLR 2025 Poster (2025)
Keywords: temporal event spotting
fine-grained video understanding
video analytics
Primary area: datasets and benchmarks
TL;DR: We propose a new benchmark and a method for analyzing fast, frequent, and fine-grained events from videos.

Abstract: Analyzing Fast, Frequent, and Fine-grained ($F^3$) events presents a significant challenge in video analytics and multi-modal LLMs. Current methods struggle to identify events that satisfy all the $F^3$ criteria with high accuracy due to challenges such as motion blur and subtle visual discrepancies. To advance research in video understanding, we introduce $F^3Set$, a benchmark that consists of video datasets for precise $F^3$ event detection. Datasets in $F^3Set$ are characterized by their extensive scale and comprehensive detail, usually encompassing over 1,000 event types with precise timestamps and supporting multi-level granularity. Currently, $F^3Set$ contains several sports datasets, and this framework may be extended to other applications as well. We evaluated popular temporal action understanding methods on $F^3Set$, revealing substantial challenges for existing techniques. Additionally, we propose a new method, $F^3ED$, for $F^3$ event detections, achieving superior performance. The dataset, model, and benchmark code are available at https://github.com/F3Set/F3Set.

Published as a conference paper at ICLR 2025
F TOWARDS ANALYZING FAST, FREQUENT, AND FINE-GRAINED EVENTS FROM VIDEOS
Zhaoyu Liu Kan Jiang", , Murong Ma², Zhe Hou', Yun Lin', Jin Song Dong² Ningbo University 2 National University of Singapore Griffith University
Shanghai Jiao Tong University
Luzy, (etaaann@uuueeeeeee edu. sg, murongma Bu. mus .edu
hou@griffith. wewriitheeuuaaae ed, lin_yunesjte edu co, dcadjs@nus sttijjsnnuseedu.... .du..g
ABSTRACT
Analyzing Fast, Frequent, and Fine-grained (F²) events presents a significant challenge video analytics and multi- LLMs. Current methods struggle identify events that satisfy all the f criteria with high accuracy due to challenges such as motion blur and subtle visual discrepancies. To advance research video understanding, we introduce Set, benchmark that consists of video datasets for precise F event detection. Datasets Set are characterized by their extensive scale and comprehensive detail, usually encompassing over 1,000 event types with precise timestamps and supporting multi-level granularity. Currently Fet contains several sports datasets, and this framework may be extended to other applications as well. We evaluated popular temporal action understanding methods on F revealing substantial challenges for existing techniques. Additionally, we propose a new method, F'ED, for F3 event detections, achieving superior performance. The dataset, model, and benchmark code are available at ht htsp:: / //github. com/ Faooo/F33ett 3Set/F3SSet
INTRODUCTION
Recognizing sequences of fast (fast-paced), frequent (many actions in a short period), and fine-grained (diverse types) events with precise timestamps (with a tolerance of 1-2 frames) is challenging problem for both current video analytics methods and multi-modal large language models (LLMs). Despite advances in fine-grained action recognition [31; 58; 51], temporal action localization [60; 6; 40; 59], segmentation [67; 33; 71; 2], and video captioning [63; 56; 49; 36), limited focus has been focused on this problem. This task is critical for various real-world applications, such sports analytics, where action forecasting [21; 65], strategic and tactical analysis [44; 45; 46; 48], and player performance evaluation [10: 55J depend on understanding detailed of event sequences. Other examples include industrial inspection [42], crucial for detecting subtle irregularities in high-speed production lines to ensure quality and safety; computer vision in autonomous driving [25], essential for accurate and instantaneous vehicle control and obstacle detection; and surveillance [53], important for the precise identification of abnormal sudden events to enhance security. However, existing methods and datasets foundational to their development only partially address the F scenario.
To facilitate the study of F events understanding, we propose a new benchmark, Set,, for precise temporal events detection and recognition. Set datasets usually have a large number of event types (on the order of 1,000), annotated with exact timestamps, and offer multi-level granularity to capture comprehensive event details. Although F3 is general problem, creating such a dataset requires domain-specific knowledge for labeling and processing: thus, in this paper, we use tennis as case study. We also introduce general annotation pipeline and toolchain to support domain experts in creating new F3 datasets. Using this pipeline, we have also been building datasets for table tennis and badminton, and a community of users is actively expanding these with other applications.
Unlike other video analysis tasks, tennis actions are characterized by their rapid succession and diversity, illustrated in Figure 1. Understanding detailed event attributes such as shot direction, technique, and outcome is essential. For example, analyzing patterns in serve directions (e.g.. "T"", "body", "wide", defined in Appendix B) or success rates can reveal players' habits and skill levels, offering strategic insights for competitive advantage [15]. This detailed analysis supports coaches and players in developing tailored strategies against different opponents [16; 47]. However, detecting

---

Published as a conference paper at ICLR 2025
Video Clip
Phae
Gevennt
nost-oourt
Gesannttt
deace
middle
Exents
hackhand
bucchand
towards
down middle
cooss court
down
cross court
volley
oppecech the not
in bound
in bound
in bound
in bound
wirner
Figure 1: Example of detecting fast, frequent, and fine-grained events with precise moments.
Fd events from videos poses sigmificant challenges, such as subtle visual differences, blurring, and the need for precise event localization. Current video understanding methods inadequately equipped to address these challenges. For instance, traditional fine-grained action recognition [9; 58; 28] assigns a single label an entire video rather than identifying sequence of events. Temporal action localization (TAL) and temporal action segmentation (TAS) often depend on pre-trained or modestly fine-tuned input features [39; 14], which lack the specificity required to capture the subtle and temporal precision. Some studies [24; 36; 43] attempt to address these issues through dense frame sampling and end-to-end training. However, this makes targeted events temporally sparse (e.g. only few events over hundreds of consecutive frames). As a result, long-term temporal correlation modules on dense visual features struggle capture event-wise causal correlations effectively.
visual details necessary for recognizing diverse events with
Moreover, Large Language Models (LLMs) [54; 61; 38] have expanded their capabilities to include multi-modal inference, encompassing text, visuals, and audio. Recognizing the potential, we con- ducted preliminary experiments on Set using GPT-4 and observed that it understood basic video contexts, such as sports types, contextual information (e.g., court type and scoreboard), and simple actions. However, it struggles with understanding F events and temporal relations between frames (e.g., shot directions). See Appendix A for details. Consequently, GPT-4 yields poor results compared to the other methods for Fd problems, and we do not use it in the experiment. By introducing Set, we hope it can help advance multi-modal LLM capabilities in F video understanding in the future.
Leveraging Set, we extensively evaluate existing temporal action understanding methods, aiming to reveal the challenges of F event understanding. To provide guidelines for future research, we conduct a number of ablation studies on modeling choices. Addressing the shortcomings of existing methods, we also propose a simple yet efficient model, F'ED, that designed for F event detection tasks and can be trained quickly on a single GPU. It outperforms existing models and can serve as baseline for further development.
Contributions.
The key contributions of this paper are as follows:
We create Sett new benchmark with datasets that feature over 1,000 precisely timestamped event types with multi-level granularity, designed to challenge and advance the state-of-the-art in temporal action understanding.
We introduce general annotation toolchain that enables domain experts to create new F3 datasets.
We propose an end-to-end model named F`ED, which can accurately detect F3 event sequences from videos through visual features and contextual sequence refinement on single GPU.
We assess the performance of leading temporal action understanding methods on F through comprehensive evaluations and ablation studies and analyze the results.
RELATED WORK
Existing F related datasets. Although datasets have been developed for temporal action under- standing, few focus on the F8 events. Table I compares existing datasets with F3Set by scale ("# Vid",, "# Clips") and characteristics like action speed ("Evt. Len."), frequency ("Evt. / sec"), ,, and gramularity ("# Classes"), which correspond to "fast", "frequent" and "fine-grained" respectively. Datasets such as THUMOS14 [27] and Breakfast [30] focus on coarse coarse-grained actions, where background context provides clear cues, and actions span seconds to minutes. In contrast, FineAction [41] and ActivityNet [4] cover a wide range of daily activities with diverse action categories, while FineGym [58] delves into detailed action types within gymnastics. Like FineGym, Set emphasizes domain-specific
2

---

Published as a conference paper at ICLR 2025
Table Comparison of existing F related datasets and Set. "Evt. Len." is the average duration of each event, and "# Evt. see" is the average number of events per second.
Datasets
Vid.
Clips.
Avg. Clip Len.
Classes
Evt. Len,
# Evt. sec
(a) Fine-grained
FincAction [41]
16,732
149.5s
101
6.9s
0.3
ActivityNet [4]
19,994
116.7s
200
49.2s
0.01
FineGym [58]
303
32,697
50,3s
530
1.7s
0.3
(b) Fast
CCTV-Pipe [42]
575
575
549.3s
16
< 0.1s
0.02
SoccerNetV2 [11]
9
9
99. 99.6min
12
< 0.1s
0.3
(c) Frequent
FineDiving [69]
135
3,000
4.2s
29
1.18
~
(d) Fast & Frequent
ShuttleSet [66]
44
3,685
10.9s
18
< 0.1s
~
Pa ANet [3]
200
2,721
360.0s
< 0.1s
~2
(d) Fast & Frequent & Fine-grained
Set
11,584
8.4s
1,108
< 0.1s
~
granularity with subtle visual differences but encounters additional challenges due to faster and more frequent actions. Besides, unlike FineGyym's typical single-player focus, Set (e.g.. tennis) features two players and a fast-moving ball, with both players rapidly moving across the court, occupying only small portions of the scene, thus increasing task difficulty. CCTV-Pipe [42] targets temporal defect detection in urban pipe systems, providing single-frame annotations for rapid event detection, though is limited in frequency and event types. Research in the sports domain has explored the detection of fast and frequent actions. FineDiving [69] segments diverse diving events, while ShuttleSet [66] and p2 ANet [3] focus on identifying strokes in fast-paced racket sports. Volleyball [26] and NSVA (basketball) [68] focus on team sports understanding and video captioning, while SoccerNetV2 [11] ball action spotting task focus on identifying the timing and type of ball-related actions. However, these datasets typically cover coarser event types and are limited to specific F aspects.
In contrast, our proposed Set is characterized by 1) rapid events that occur instantaneously. 2) high frequency of approximately one event per second, and 3) extensive granularity with a larger number of detailed event classes. These attributes introduce novel challenges.
F8 event understanding Detecting F events poses unique challenges due to their rapid temporal dynamics, high occurrence rates, and subtle visual distinctions, requiring precise temporal and contextual understanding. Fine-grained action detection has been explored in tasks covering diverse daily activities [4; 41], using features extracted by video encoders pre-trained on datasets like Kinetics- 400 [29] and a detection head for classification. However, such pre- pre-trained extractors often miss domain-specific nuances. Domain-specific methods in FineGym [58] and FineDiving [58] utilize end-to-end training to incorporate domain knowledge. These methods often encode videos into non-overlapping snippets downsample frames, yielding coarse temporal features insufficient for detecting fast-paced events spanning only 1-2 frames. Related works such as ShuttleSet [66] and p2 [3] address fast and frequent event detection in racket sports by employing end-to-end models that extract frame-wise features and use detection heads (e.g.. BMN [37] or GRU [8]) to classify each frame. To address class imbalance, the loss weight of the foreground classes set higher than the background during training [24]. While these approaches achieve precise temporal spotting, their scalability larger action classes is limited by challenges like long-tail class distributions and inadequate modeling of event-wiss correlations. Our proposed FSED overcomes these issues through frame-wise dense processing, multi-label classification head to handle minor event differences and class imbalances, and a contextual module to refine predictions by leveraging event-wise causal relationships, enhancing both precision and robustness in F3 event detection.
3 SET:: A BENCHMARK DATASET FOR F3 EVENT DETECTION
Recognizing the limitations existing video datasets for F event understanding, we introduce Set, new benchmark for precise temporal F3 events detection and recognition. Given the need for
3

---

Published as a conference paper at ICLR 2025
Setet
Badminton
Tennis Double
Tenmis Single
Table Tannis
SC
SC2
SC3
Sc4
SCS
505
SC7
SCB
near
far
deuce
ad
mid
ffr
Ihh
serve
return
strake
10
Tob
appr
Fi
uif
for deuce_- -_eevve_ T_o______ppprp -_arr_n
slice -_/e
Sub-closs
Element
Video Clip
Figure 2: Breakdown of F Set event class annotation.
domain-specific expertise in creating F datasets, this section uses tennis as a case study to illustrate Set's event descriptions, construction process, and key properties. We also propose a general annotation pipeline and toolchain that empowers domain experts to develop new F datasets for diverse applications. Applying the same approach, we have also built F datasets for other domains, including tennis doubles, badminton, and table tennis (see link).
3.1
SET EVENT DESCRIPTION
We use tennis illustrate F event descriptions, introducing key lexicon and defining F events. Datasets have been built for other F' d'dmainss, including tennis doubles, badminton, and table tennis, with similar event definitions. Details are Appendix
Lexicon. A tennis court is divided into deuce, middle, and ad regions. The initial shot, a "serve," " targets the T, Body (B), or Wide (W) areas. A "return" follows if the receiver's shot lands in bounds. Subsequent shots, or "sttrkke",, can directed "cross-court" (CC), "down the line" (DL), "down the middle" (DM), "inside-in" (II), or "inside-out" (IO) using either "forehand" (fh) or "backhand" (bh). Players may "approach" (apr) the net on shorter balls. Shot techniques include "ground stroke/top spin" (gs), "slice", "volley", and "lob", with outcomes: "in-bound", "winner", "forced error", or "unforced error". More detailed definitions can be found in Appendix B.
F events. Formally, each event consists of 8 sub-classes, denoted as sC1, 802, 508:
8C1 hit by which player: (1) near- or (2) far-end player:
8C2 hit from which court location: (3) deuce, (4) middle, or (5) ad court:
SC3 hit at which side of the body: (6) forehand or (7) backhand;
8C4 shot type: (8) serve, (9) return, (10) stroke;
8C5 shot direction: (11) T, (12) B, (13) W. (14) CC, (15) DL. (16) DM, (17) II, or (18) IO;
8Cg shot technique: (19) (20) slice, (21) volley. (22) lob, (23) drop, (24) smash;
8C7 player movement: (25) approach:
8C8 shot outcome: (26) in, (27) winner, (28) forced error, or (29) unforced error.
Altogether, there are 29 elements and 1,108 event types based on various combinations (Figure 2).
Similarly, for other domains, badminton contains 6 sub-classes, 28 elements and 1008 event types; table tennis contains 7 sub-classes, 23 elements and 1296 event types; and tennis doubles contain 26 elements and 744 event types. Compared to existing racket sports video datasets [3; 66], Set offers additional dimensions, such as shot direction and outcomes, which are crucial for identifying playing patterns and success rates. Please refer to Appendix C. for more details.
3.2 T SET DATASET CONSTRUCTION
Video collection. For tennis, we collected publicly available high-resolution singles matches (2012-2023) from YouTube, including Grand Slams, Olympics, and major ATP/WTA tournaments. The dataset includes various court surfaces (hard, clay, grass), male and female players, and both right- and left-handed competitors. These videos feature complete rallies, match footage, and detailed player data. Similar criteria were used for tennis doubles, badminton, and table tennis videos.
4

---

Published as a conference paper at ICLR 2025
Event Labeling
Court Position
deuce
mid
For
Nearad
Near mid
Near deuce
Beckhand
Shot Type
Serve
Return
Stroke
DIAUP
Shot Technique
Ground Stroke
Siice
Xcoo
Velley
Lob
Drop
Smash
Laboled Events
Label Event
Netideo 1dt1 Tostal, Irraren: 372, "Spa": 25,
Delete Frame
Gutcome
Sarve Labels
Fee-Bir
UntoekErr
Figure 3: An interface of the labeling tool. The panel on the right is application-customizabbee
Annotation pipeline and toolchain. After data collection, we use three-stage annotation process designed to maximize automation and minimize manual effort. This pipeline is adaptable to various sports broadcast videos and broader domains:
(I) Video segmentation: The first stage is to segment full broadcast video into shorter clips using context-aware scene detector [1] that automatically identifies jump cuts within the video.
(2) Clip selection: The second stage is to select targeted clips (e.g., clips contain tennis rallies) using Siamese network compare each clip with "base image" indicative of the scene of interest.
(3) Fs event annotation: The final stage is identify the precise event moments (e.g., frames when a player hits the ball) and record the corresponding event types through an annotation tool.
The first two steps are automated and applicable to range of sports videos, facilitating the efficient breakdown of lengthy videos into relevant clips. For the final phase, we developed an interactive annotation interface, shown in Figure 3. The tool allows users to navigate clips quickly (c.g.. 1-second increments) or review them frame by frame, enabling efficient identification of key events (e.g.. hitting moments). It supports selecting shot types and identifying court positions through direct clicks the video, with each click displayed for immediate verification. Object-level detection can assist the process, and foolproof design minimizes errors from accidental clicks misjudgments. This tool is adaptable to other sports by incorporating domain-specific knowledge, broadening its applicability.
Our annotation team consists of 8 members. We provided them with specialized training and rigorous pre-tests before beginning the official annotation work, along with supporting materials such as slides and demonstrations. Each annotator was assigned an equal portion of the dataset, totaling 1,450 clips (rallies) each. The manual labeling takes roughly 30 hours to finish all 1 1,450 clips. Following the initial annotation phase, we conducted multiple rounds of cross- validation involving random sampling of rallies and quality checks among annotators to ensure the accuracy of the event-based labels. In cases where conflicting annotations arose, annotators were asked to input the labels they believed to be correct. The final label was determined based on majority vote among the annotators.
3.3 SET DATASET STATISTICS AND PROPERTIES
Key statistics for Set tennis dataset summarized in Table 2. Statistics for other FD datasets, including badminton, table tennis, and tennis doubles, are provided in the Appendix D. We employ a training, validation, and testing split of 3:1:1, with the training and validation sets drawn from the same video sources, while the test set features clips from distinct videos.
Event Timestamp. Unlike typical TAL and TAS tasks, where action spans several frames or seconds, the duration actions in racket sports is often ambiguous. Thus, stroke actions are defined instantaneous events, recording only the moment of ball-racke contact [62] shown in Figure 1.
5

---

Published as a conference paper at ICLR 2025
Table 2: Summary of Set tennis dataset statistics.
Category
Details
Category
Details
Matches
114 broadcast matches
Clips
11,584 rallies 8.4 see
Players
75 (30 men, 45 women)
Avgerage Clip Duration
Handedness
68 right-handed. left-handed 25-30 FPS
Total Shots
42,846
Frame Rate (FPS)
Shots Per Rally
to 34
Multi-level granularity. Depending the requirements of the analytics task, Fe focus subset of sub-classes, enabling flexible granularity. We define a parameter G' E (P(fsc1....... 80g }}, where P( P((5c]-- ..... 80a)) }) is the power set of {sc1:-.., sca), ) to select sub-classes and form different levels of granularity. We define 3 granularity levels using Set tennis as an example. At the coarse level, Glow {sc1, SC3, SC4, sCg J includes sub-classes, 11 elements, and 38 event types. This level captures essential but broad information. At finer level, Gmid {sc1.... scs} consists of sub-classes, 24 elements, and 365 event types. This granularity provides more detailed event representations. At the most detailed level, Ghigh = {sc1,.... se8) encompasses all 8 sub-classes, 29 elements, and 1,108 event types. This level ideal for precise and comprehensive event analysis. This multi-level granularity enhances F Set's flexibility for diverse real-world tasks.
3.4 ETHICAL CONSIDERATIONS
Set is constructed from publicly available sports broadcasts, ensuring compliance with ethical and legal standards. We do not redistribute video content, providing only YouTube links to maintain adherence copyright policies. The dataset focuses professional players in public tournaments, used strictly for academic research. While
avoiding private or off-court data and ensuring anonymization is not applied, these players public figures, emphasize that the dataset should not be used for non-research purposes. A more detailed discussion on privacy, consent, and bias mitigation provided in Appendix E.
OUR PROPOSED APPROACH: FED
Acknowledging the challenges and limitations of existing approaches, we propose a simple yet effective method named Fast Frequent Fine-grained Event Detection network (FED), illustrated in Figure 4. It is designed for F event detection and can serve as a baseline for further development.
Problem formulation. Let X E "X denote the input, consisting of N RGB frames of size H x W. The output is a sequence of M event-timestamp pairs ((Fg,t) ((F,,t))...... (EM, fM)) )), where Eg is the event type with C' classes and is the corresponding timestamp for i E (1, (1...., M). Additionally, each event E, can also be expressed as a vector (ei, Gi with each element eLj E 1) indicating the presence or absence the jth element in event E,, where j is an integer j (1, (1,..., K). The parameter K, which defines the number of elements in each event vector.
Video Encoder (VE). The first stage of both baselines and our model will extract spatial-temporal frame-wise features. The video encoder (VE) consists of visual backbone, followed by bidirec- tional GRU capture long-term visual dependencies: F Femb VE(X), with F Femb Ndd' xd'
Event Localizer (LCL). Utilizing the frame- wise features F Fmmb. the event localizer (LCL) employs a fully connected network with a Sigmoid activation function to perform dense binary classification, aiming accurately identify specific event instances. For N-frame clip, the output is represented as (f.....,pN)) where each p. denotes the probability that an event occurs at the corresponding timestamp: (pI,..., : PN)
(F Ground truth labels (pI,..., ,,.,... with p (0, 1) are used to compute the discrepancy between the predicted probabilities and the actual values using binary cross-entropy loss as: LLCL = * DNNIP log(p.) + (1 pi) log(1 p.).
Multi-label Event Classifier (MICC.. Upon detecting events, we proceed to categorize them into specific types using multi-label classification module (MLC). This module, a fully connected network, takes the identified event features f; from F Femo as inputs to predict the event types: [ê.. ē.,,), where K denotes the number elements, fr represents
É, = Sigmoid(MLL(()) the features for the event at the 8th frame, É, is the predicted event type, and é.,j [o, 1] is the probability of É, containing the jth element. For video clip with M events, the ground truths

---

Published as a conference paper at ICLR 2025
Event Localizer (L.CL)
Contentual Module (CTX)
(B1,
Video Encoder (95)
(2)
(Bar
f fg Car-1 Eat
(arr-1)
Clip
Multi-Label Classifier (MLC)
(Bar,far)
8
8g
Ba
Figure 4: Overview of FED. RGB images are processed by VE to capture frame-wise spatial- temporal features, which are passed I.CL identify event timestamps and MI. to predict labels. Outputs from LCL and MLC are combined ('plus' symbol) to form an event representation sequence and refined by CTX module. 'Red squares' represent errors from purely visual predictions.
given as (Ei...., EM) with each E; represented as vector of K elements [ei, C. The loss can be represented by LMLC- = 1 ( E eij log((e.)) + (1 (1.) log(1 -
Contextual module (CTX) Video encoders often struggle to extract insightful visual features from fast-paced videos due to motion blur, and objects of interest, such as players, may only occupy a small portion of the frame. This result in the loss of crucial visual details for fine-grained action classification, particularly when resizing images to 224 x 224. Selecting the best- best-predicted event types naively might, therefore, produce invalid event sequences. To address this, introduce a contextual module (CTX), designed to concurrently learn contextual knowledge from event sequences during end-to-end training: (IE,..., (M) = CTX(É)..... EM). CTX employs a bidirectional GRU to process the predicted event sequence É and outputs a refined sequence E, [ee,...., @x), integrating both visual-based predictions and contextual correlations across events. The loss is calculated for each refined event: LCTX = M EEM( ( Σ/ Ci.s log(er,) + (1 - -e.) log(1 - e,,5))
EXPERIMENTS
In this section, we benchmark existing temporal action understanding methods, including TAL, TAS, and TASpot, on the Set dataset and conduct series of ablation studies.
Evaluation metrics. The evaluation metries used in work carefully chosen compre- hensively assess both the temporal precision and classification accuracy of detected events, which are critical for F$ event detection. These metries align with evaluation standards in similar tasks. Edit Score [32] measures the similarity between predicted and ground truth event sequences using Levenshtein distance, capturing errors event sequence structure, such as missing, additional, or misordered events. This metric is particularly valuable for evaluating models where the temporal order and completeness of event sequences are essential [23]. Mean F1 Score with Temporal Tolerance evaluates both classification and temporal localization accuracy [24; 23]. By considering prediction correct only when its timestamp aligns within a strict temporal tolerance (c.g.. #1 frame) and its class correctly identifies, this metric ensures that models are assessed on their ability to achieve precise temporal spotting alongside accurate classification. Given the long-tail distribution of event types in the dataset, where some events are extremely rare, report two variants of the mean F1 score to ensure a balanced evaluation: F1 Flevt, the average Fl score across all event types, and F Flemm. the average Fl score across all elements, which typically presents more balanced distribution.
Baselines. Existing temporal action understanding frameworks typically incorporate two key components: a video encoder for visual feature extraction and a head module for specific tasks such as detection or segmentation. Applying these models directly to our study presents challenges, as they generally utilize a two-stage training process- employing static, pre-trained video encoder for feature extraction and training only the head module. This approach often fails to capture fine- mne-grained, domain-specific events due to its reliance on temporally coarse, non-overlapping, or downsampled video segments. To address these limitations, we have adapted these temporal action understanding methods to develop new baselines better suited for detecting F3 events. Given the rapid pace and short duration of tennis shots, it is crucial to utilize frame-wise feature extraction [7] (discussed in Section 5.2). Besides, end-to-end training with video encoder fine-tuning is required to capture the subtle event differences. Moreover, the classification of some sub-classes (e.g., shot direction, outcome) demands long- term temporal reasoning to integrate information from subsequent frames.
Consequently, we focus on established feature extractors: TSN [64], SlowFast [20]. I3D [5], VTN [52], and TSM [35], which enable frame-wise feature extraction and end-to-end training. We pair
7

---

Published as a conference paper at ICLR 2025
Table 3: Experimental results on Set (tennis) with 3 levels of granularity. Full table in Appendix G.
Set (Gragh)
Set (Gmss)
Set (Giow)
Video encoder
Head arch.
Fleet
Flelm
Edit
Fleve
el
Edit
Flact
Flelm
Edit
TSN [64]
MS-TCN [19]
15.9
59.8
53.5
23.2
60.9
65.8
45.7
70.4
72.8
ActionFormer [72]
18.4
60.6
55.2
24.8
61.9
67.3
48.7
70.6
72.2
E2F-Spot [24]
24.7
65.3
60.1
31.5
66.2
71.0
53.5
73.6
75.0
SlowFast [20]
G-TAD [70]
23.0
66.1
64.0
29.6
66.5
74.2
53.3
76.0
77.9
ActionFormer [72]
28.7
70.0
67.6
35.5
70.9
76.4
59.3
77.1
81.5
E2E-Spot [24]
25.9
69.4
65.7
33.8
70.4
75.4
55.5
76.5
79.5
I3D [5]
S2F-Spot [24]
22.7
59.7
68.7
27.1
60.7
74.2
51.9
67.7
78.3
VTN [52)
E2E-Spot [24)
14.8
58.3
56.7
20.0
59.4
68.2
39.7
63.1
73.1
TSM [35]
MS-TCN [19]
21.7
67.3
58.6
30.4
69.5
73.0
50.2
74.0
75.3
ASformer [71]
17.6
61.9
57.5
25.5
64.0
74.2
46.0
72.9
74.0
G-TAD [70]
16.9
62.5
55.2
29.8
66.9
74.8
39.8
70.1
67.2
ActionFormer [72]
22.4
65.7
60.3
31.0
68.2
74.7
52.4
73.8
74.9
E2E-Spot [24]
31.4
71.4
68.7
39.5
72.3
77.9
60.6
78.4
82.1
TSM[35]
F'DD
40.3
75.2
74.0
48.0
76.5
82.4
68.4
80.0
87.2
each encoder with five representative head module architectures from existing methods: MS-TCN [19] and ASFormer [71] from TAS, G-TAD [70] and ActionFormer [72] from TAL and E2E-Spot [24] from TASpot, to establish a set of new baseline models for our study. To identify hitting moments and their respective event types, frame-wiss wiie dense multi-class classification is applied to identify each frame as either background or one of the event types.
Implementation details. We implement and train models on FSttt in an end-to-end manner. The video encoder takes video clip X down-scaled and cropped to 224 x 224 to extract frame-wise visual features. Subsequently, each head module processes per- per-frame features to identify a sequence of F3 events and their timestamps. For implementation details, please refer to Appendix F.
5.1 RESULTS AND ANALYSIS
The evaluation results presented in Table provide several critical insights into the performance of various methods across different levels of granularity (Glows Gmidd and Ghigh). A general trend emerges where performance decreases as granularity increases, underscoring the growing challenges associated with finer granularity. While certain methods demonstrate some robustness, the overall efficacy across all approaches remains suboptimal, particularly at higher levels of granularity, indicating the challenge of precise F3 event detection task.
Simple 2D CNNs (e.g. TSN), which process frames independently, are inadequate for F event detection due to their inability to capture critical spatial-temporal correlations between frames. Lacking temporal modeling, they struggle to distinguish visually similar events, resulting in poor performance, especially at higher granularity levels. Advanced video encoders such as I3D [5], SlowFast [20], and transformer-based VTN [52], which excel in other video understanding tasks, face significant challenges with FSSet. These models process video data using techniques like non- overlapping snippets frame downsampling, resulting in coarse temporal features. While effective for long-duration actions, such approaches struggle to detect the rapid, short-duration events in F3, which rely on precise temporal cues spanning only 1-2 frames. This suggests that increasing video encoder complexity does not necessarily improve performance for fast-action detection in Set... Notably, simpler models like TSM, paired with advanced 2D CNNs such RegNet-Y Y [57], outperform these complex encoders. This highlights the importance of capturing subtle visual differences over short temporal spans, demonstrating that the ability to extract one-ggained temporal cues is more impactful than model complexity.
Head modules such as transformer-baas based ActionFormer, and GaU-based E2E-Spot, generally outper- form other methods. This advantage highlights their effectiveness in capturing long-term temporal dependencies through end-to-end training. Notably, E2E-Spot consistently outperforms FotionFormer across most settings, suggesting that GRU-based architectures may offer advantageous trade-off between efficiency and representational power for certain types of temporal correlations.
Our proposed F'ED model, leveraging the TSM video encoder, achieves the best performance among all granularity levels. This is attributable to two key design choices: the multi-label classifier and the contextual module. Detailed discussions of these design elements presented in the next section.
8

---

Published as a conference paper at ICLR 2025
Table 4: Ablation and analysis experiments. The default model takes stride size 2 and clip length 96.
Set (Griga)
Set (Gmus
SSe (Glow)
Experiment
Fladm
Edit
Flas
Fl
Edit
Flaur
el
Edit
TSM E2E-Spot
31.4
71.4
68.7
39.5
72.3
77.9
60.6
78.4
82.1
(a) Feature extractor
13D [S] (clip-wise)
22.7
59.7
68.7
27.1
60.7
74,2
51.9
67.7
78.3
VTN [52] (video transformer)
14.8
58.3
56.7
20.0
59.4
68.2
39.7
63.1
73.1
ST-GCN++ [17] (skeleton-based)
25.4
62.1
56.1
32.4
63.9
63.5
55.1
69.4
73.2
PoseConv3D [18] (skelelon-based)
20.1
54.5
53.2
26.0
55.4
61.9
48.8
63.0
69.7
(b) Stride size 4 Stride size 8
25.9
69.2
62.7
33.4
69.9
73.0
60.0
77.9
78.8
14.0
56.7
44.3
18.5
57.4
54.8
40.4
67.0
59.2
(c) without GRU
27.6
69.0
60.6
38.0
71.3
75.3
54.7
74.1
73.4
(d) Clip Length =
26.3
67.4
54.5
35.5
69.4
71.8
53.2
75.1
68.9
Clip Length
30.7
71.2
67.4
38.6
72.4
77.5
58.4
77.9
81.1
Clip Length 192
29.3
70.3
65.7
37.3
71.4
77.0
58.8
77.1
80.4
(e) Multi-label
37.9
74.3
45.9
75.6
80.1
66.6
80.1
85.1
(1) Multi-labet + CTX (Transformer)
39.0
74.3
72.8
50.5
75.5
81.8
63.4
79.6
86.8
Multi-tabel + CTX (BiGRU)
40.3
75.2
74.0
48.0
76.5
82.4
68.4
80.0
87.2
5.2 ABLATION STUDY
We selected the highest-performing baseline model (TSM + E2E-Spot) as our default configuration for the subsequent ablation studies. More ablation studies can be found in Appendix H.
Feature extractor. An effective feature extractor crucial for accurate F event detection. Be- low, we summarize some key findings (details in Appendix H). (1) Frame-wise feature extraction outperforms clip-wise methods, which divide inputs into non-overlapping segments. Experiments show clip- wise methods produce temporally coarse features and hinder precise event detection. (2) Transformer-be -based video encoders such as VTN [52] struggle on Set due to high computational costs and limited ability to effectively capture short-term temporal correlations. (3) In addition to RGB inputs, we also experimented with kkeleton-based pose estimation methods, including +T-GCN++ [17] and PoseConv3D [18] with human key points input. While they excel in efficiency and interpretability, they lack critical details like shot direction, limiting performance on Sett.
Sparse sampling. Increasing the stride size allows for a broader temporal coverage within a fixed sequence length. This sparse sampling technique is prevalent in many video understanding tasks [40; 34), offering high efficiency and reasonable accuracy. However, this approach proves inadequate for task, where events are characterized by their rapid occurrence, frequency, and fine granularity. As illustrated in Table 4(b), increasing the stride size to 4 and 8 leads to marked decline in performance, underscoring the importance of dense sampling for detecting F3.
Long-term temporal reasoning. The default model employs spatio-temporal video encoder (TSM), complemented by a bidirectional Gated Recurrent Unit [13] (GRU) head for enhanced long- term temporal integration. To assess the necessity of long-term temporal reasoning, we replaced the GRU module with fully connected layer. The results, presented in Table 4(c), indicate significant performance decline relative to the original configuration. This finding highlights the essential role of long-term temporal reasoning in analyzing sub-classes such shot direction, outcomes, and player movements that require information from subsequent frames.
Clip length. The sensitivity of sequence models to varying input clip lengths, which encapsulate different temporal contexts, is notable. In Sett, the incidence of F3 events correlates directly with clip length. Table 4(d) shows that shorter clips result in fewer events per sequence, hindering the model's ability to leverage long-term dependencies among consecutive events effectively, Conversely, while longer clip lengths yield improved results, the marginal gains diminish with increasing length.
Multi-class versus multi-label classification. The challenge of modeling over 1,000 possible event type combinations as multi-class classification problem formidable. For example, consider two events, E, (far bh_.aadddhtssseee stroke DL_slice sesliice_pr_iin apr_in) and Eg (far_ad www.stooeeeDL....... .roo lil_droppapr_ii an)_ which differ only in shot technique (slice vs. drop). Although similar, multi-class classification treats these as distinct classes, thus reducing training efficiency and exacerbating the long- tail distribution bias towards more frequent classes. more natural approach is multi-label classification, where each event can belong to multiple sub-class elements (e.g., ['far", 'ad', 'serve", 'W", 'in')). Thus, E1 and

---

Published as a conference paper at ICLR 2025
Table 5: Experimental results on other "semi- Feemi-F3r-aa datasets.
ShuttleSet [66]
FineDiving [69]
FineGym [58]
SoccerNet [11]
CCTV-Pipe [42]
Head arch.
Flast
Edit
Flavi
Edit
Flaet
Edit
Fleat
Edit
Flaet
Edit
MS-TCN [19]
70.3
74.4
65.7
92.2
57.6
65.3
43,4
74.5
25.8
31.3
ASformer [7]]
55.9
70.6
49.9
87.6
53.6
66.3
46.3
76.
15.4
33,4
G-TAD [70]
48.2
61.1
52.1
82.6
45.8
51.4
42.3
72.3
31.3
33,6
ActionFormer [72]
62.1
67.5
68.3
92.4
54.0
59.7
43,0
64.6
18.8
29.5
E2E-Spot [24)
70.2
75.0
75.8
93.7
62.1
65.4
46.2
72.9
27.2
35.2
FEED
70.7
77.1
77.6
95.1
70.9
70.7
48.1
76.6
37.0
39.5
E2 only differ in shot technique but are identical in other aspects. This adjustment facilitates more effective training and shows an increase in performance, as shown in Table 4(c).
Contextual knowledge. Beyond the statistical results Table 3, analysis of predicted event sequences reveals that current baselines may produce invalid sequences due to logical errors or uncommon practices. For instance, a right-handed player cannot logically direct a forehand shot from the deuce court as "II" or "IO". Similarly, an event ending in winner or eπror should logically conclude the sequence. Additionally, it is uncommon for a player to hit with backhand when the ball is played their forehand side. Further examples are detailed in Appendix I. These observations indicate that existing baselines fail to effectively capture event-wise contextual correlations. By adding the CTX module, the performance further increases as shown in Table 4(f). We also compared BiGRU and Transformer Encoder for the CTX module. BiGRU performed slightly better, likely due its efficiency in modeling short event sequences (usually 20 per clip) with fewer parameters.
5.3 GENERALIZABILITY TO "SEMI-F3" DATA
F$ task possesses broad applicability across numerous real-world domains, such as sports, autonomous driving, surveillance, and production line inspection. Nevertheless, creating such a F dataset necessitates substantial expertise and extensive labeling efforts. We have found that existing video datasets often fail to fully address all three dimensions of the F3 grained". In this section, we conducted experiments on several "semi-F3" datasets that partially meet these criteria, including Shuttleset [66] for badminton (racket sport), FineDiving [69] for diving (individual sports), FineGym [58] for gymnastics (individual sports), SoccerNetV2 [50] (team sports), and CCTV-Pipe [42] for pipe defect detection (industrial application). We report only the Flevt and Edit score, as not all datasets necessitate multi-label classification given their limited event types. For the video encoder, we chose TSM, which consistently outperforms the others on average.
"frequent", and "fine-
Performance across different domains can vary significantly depending on the difficulty of tasks and the scale of datasets. For instance, the CCTV-Pipe dataset, targeting temporal defeet localization in urban pipe systems, shows suboptimal performance due to factors such as ambiguous single- frame annotations for each defect, multiple defects at the same time, long-tailed distribution of defect types, and limited dataset size. Our performance is better than the results reported in [42]. Generally, methods that effectively handle Set tend to perform well across other applications, as indicated in Table 5. Our ECDD outperforms existing baselines in all datasets, demonstrating its robust generalizability for detecting "semi-F3" events across various domains. While F't event detection benefits from accurate event localization, a high-performing LCL module is not a hard prerequisite (see Appendix J). Therefore, our method can be generalized and benefit broader applications.
CONCLUSION AND FUTURE WORK
In this study, we addressed the challenge of analyzing fast, frequent, and fine mne-grained (Fo) events from videos by introducing Set, a benchmark for precise temporal F3 event detection. S datasets usually feature detailed event types (approximately 1,000), annotated with precise timestamps, and provide multi-level granularity. We have also developed a general annotation toolchain that enables domain experts to create F datasets, thereby facilitating further research in this field. Moreover, we proposed FSED, an end-to-end model that effectively detects complex event sequences from videos, using combination of visual features and contextual sequence refinement. Our comprehensive evaluations and ablation studies of leading methods in temporal action understanding on F highlighted their performance and provided critical insights into their capabilities and limitations. Moving forward, we aim to extend the scope of F3 task to more real-world scenarios and advance the development of F3 video understanding.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research supported by the Singapore (AISG3-RP-2022 Any opinions, findings and conclusions or reconmendations expressed in this material are those of the author(s) and do not reflect the views of funding bodies.
REFERENCES
[1] Pyscenedetect. https://...seeeeee...
[2] Nadine Behrmann, S Alireza Golestaneh, Zico Kolter, Juergen Gall, and Mehdi Noroozi. Unified fully and timestamp supervised temporal action segmentation via sequence to sequence translation. In European Conference on Computer Vision, pp. 52-68. Springer, 2022.
[3] Jiang Bian, Xuhong Li, Tao Wang, Qingzhong Wang. Jun Huang, Chen Liu, Jun Zhao, Feixiang Lu, Dejing Dou, and Haoyi Xiong. P2anet: large-scale benchmark for dense action detection from table tennis match broadcasting videos. ACM Transactions on Multimedia Computing, Communications and Applications, 20 2024.
[4] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference computer vision and pattern recognition, pp. 961-970, 2015.
[5] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In The IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
[6] Yu-Wei Chao, Sudheendra Vijaannarasimhan. Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar. Rethinking the faster r-cnn architecture for temporal action localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1130-1139, 2018.
[7] Minghao Chen, Fangyun Wei, Chong Li, and Deng Cai. Frame-wise action representations for long videos via sequence contrastive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 13801-13810, 2022.
[8] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 2014.
[9] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitcher dataset. In Proceedings of the European conference on computer vision (ECCV), pp. 720-736, 2018.
[10] Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. Actions speak louder than goals: Valuing player actions in soccer. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1851-1861, 2019.
[11] Adrien Deliege, Anthony Cioppa, Silvio Giancola, Meisam Seikavandi, Jacob V Dueholm, Kamal Nasrollahi, Bernard Ghanem, Thomas B Moeslund, and Mare Van Droogenbroeck. Soccernet-v2: A dataset and benchmarks for holistic understanding of broadcast soccer videos. In The IEEE/CVF conference computer vision and patlern recognition, pp. 4508-4511 2021.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp.
Icee, 2009.
[13] Rahul Dey and Fathi M Salem. Satianariants of gated recurrent unit (gru) neural networks. In 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS), pp. 1597-1600. IEEF, 2017.
11

---

Published as a conference paper at ICLR 2025
[14] Guodong Ding, Fadime Sener, and Angela Yao. Temporal action segmentation: An analysis of modern techniques. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023,
[15] Jin Song Dong, Ling Shi, Kan Jiang, Jing Sun, et al. Sports strategy analytics using probabilistic reasoning. 2015 20th International Conference on Engineering of Complex Computer Systems (ICECCS), pp. 185. IEEE, 2015.
[16] Jin Song Dong, Kan Jiang, Zhaoyu Liu, Chen Dong, Zhe Hou, Rajdeep Singh Hundal, Jingyu Guo, and Yun Lin. Sports analytics using probabilistic model checking and deep learning. In 2023 27th International Conference on Engineering of Complex Computer Systems (ICECCS), pp. 7-11. IEEE, 2023.
[17] Haodong Duan, Jiaqi Wang, Kai Chen, and Dahua Lin. Pyskl: Towards good practices for skeleton action recognition. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 7351-7354, 2022.
[18] Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. Revisiting skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2969-2978, 2022.
[19] Yazan Abu Farha and Jurgen Gall. Ms-ten: Multi-stage temporal convolutional network for action segmentation. In The IEEE/CVF conference on computer vision and pattern recognition, pp. 3575-3584, 2019.
[20] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In The IEEE/CVE international conference on computer vision, pp. 6202-6211, 1, 2019.
[21] Panna Felsen, Pulkit Agrawal, and Jitendra Malik. What will happen next? forecasting player moves in sports videos. In Proceedings of the IEEE international conference on computer vision, pp. 3342-3351,2017.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016.
[23] Yuchen He, Zeqing Yuan, Yihong Wu, Liqi Cheng, Dazhen Deng, and Yingcai Wu. Vis- tec: Video modeling for sports technique recognition and tactical analysis. arXiv preprint 15X5:22402.15952. 2024.
[24] James Hong, Haotian Zhang, Michaël Gharbi, Matthew Fisher, and Kayvon Fatahalian. Spotting temporally precise, fine-grained events in video. European Conference on Computer Vision, pp. 33-51. Springer, 2022.
[25] Xinyu Huang. Xinjing Cheng, Qichuan Geng, Binbin Cao, Dingfu Zhou, Peng Wang. Yuanqing Lin, and Ruigang Yang. The apolloscape dataset for autonomous driving. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp.
2018.
[26] Mostafa S Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori. A hierarchical deep temporal model for group activity recognition. In The IEEE conference on computer vision and pattern recognition, pp. 1971-1980, 2016.
[27] Haroon Idrees, Amir Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The thumos challenge on action recognition for videos "in the wild". Computer Vision and Image Understanding, 155:1-23, 2017.
[28] Kan Jiang, Masoumeh Izadi, Zhaoyu Liu, and Jin Song Dong. Deep learning application in broadcast tennis video annotation. In 2020 25th International Conference on Engineering of Complex Computer Systems (ICECCS), pp. 53-62. IEEF, 2020.
[29] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint /rXiv:/705.0000 /705.0095000 2017.
12

---

Published as a conference paper at ICLR 2025
[30] Hilde Kuehne, Juergen Gall, and Thomas Serre An end-to-end generative framework for video segmentation and recognition. In Proc. IEEE Winter Applications of Computer Vision Conference (WACV 16), Lake Placid, Mar 2016.
[31] Colin Lea, René Vidal, and Gregory D Hager. Learning convolutional action primitives for fine- grained action recognition. In 2016 IEEE international conference on robotics and automation (ICRA), pp. 1642-1649, IEEE, 2016.
[32] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks for action segmentation and detection. In The IEEE Conference Computer Vision and Pattern Recognition, pp. 156-165, 2017.
[33] Zhe Li, Yazan Abu Farha, and Jurgen Gall. Temporal action segmentation from timestamp supervision. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8365-8374, 2021.
[34] Chuming Lin, Chengming Xu, Donghao Luo, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu. Learning salient boundary feature for anchor-fres free temporal action localization. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pp. 3320-3329, 2021.
[35] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video under- standing. In The JEEF/CVF international conference on computer vision, pp. 7083-7093, 2019.
[36] Kevin Lin, Linjie Li. Chung- Chinn Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. Swinbert: End-to-end transformers with sparse attention for video captioning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17949-17958, 2022.
[37] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. Bmn: mouhdary-matching network for temporal action proposal generation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3889-3898, 2019.
[38] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances neural information processing systems, 36, 2024.
[39] Meng Liu, Liqiang Nie, Yunxiao Wang, Meng Wang, and Yong Rui. A survey on video moment localization. ACM Computing Surveys, 55(9):1-37, 2023.
[40] Xiaolong Liu, Song Bai, and Xiang Bai. An empirical study of end-to-end temporal action detection. In The JEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 20010-20019, 2022.
[41] Yi Liu, Limin Wang, Yali Wang, Xiao Ma. and Yu Qiao. Fineaction: A fine fine-grained video dataset for temporal action localization. IEFE transactions on image processing, 31:6937-6950, 2022.
[42] Yi Liu, Xuan Zhang, Ying Li, Guixin Liang, Yabing Jiang, Lixia Qiu, Haiping Tang, Fei Xic, Wei Yao, Yi Dai, al. Videopipe 2022 challenge: Real-world video understanding for urban pipe inspection. In 2022 26th International Conference on Pattern Recognition (ICPR), pp. 4967-4973. IEEE, 2022.
[43] Zhaoyu Liu, Jingyu Guo, Mo Wang, Ruicong Wang. Kan Jiang, and Jin Song Dong. Recognizing a sequence of events from tennis video clips: Addressing timestep identification and subtle class differences. In 2023 IEEE 28th Pacific Rim International Symposium on Dependable Computing (PRDC), pp. 337-341. IEEE, 2023.
[44] Zhaoyu Liu, Kan Jiang, Zhe Hou, Yun Lin, and Jin Song Dong. Insight analysis for tennis strategy and tactics. In 2023 IEEE International Conference on Data Mining (ICDM), pp. 1 1175-1180. IEEE, 2023.
13

---

Published as a conference paper at ICLR 2025
[45] Zhaoyu Liu, Chen Dong, Chen Wang, Tian Yu Dong, and Kan Jiang. Exploring team strat- egy dynamics in tennis doubles matches. In International Sports Analytics Conference and Exhibition, pp. 115- Springer, 2024.
[46] Zhaoyu Liu, Murad Durrani, Leong Yu Xuan, Julian-Frederty Simon, and Tan Yong Feng Deon. Strategy analysis in nfl using probabilistic reasoning. In International Sports Analytics Conference and Exhibition, pp. 16-128. Springer, 2024.
[47] Zhaoyu Liu, Murong Ma, Kan Jiang, Zhe Hou, Ling Shi, and Jin Song Dong. Pesp#t denotational semantics with an application in sports analytics. In The Application of Formal Methods: Essays Dedicated Jim Woodcock the Occasion of His Retirement, pp. 71-102. Springer, 2024.
[48] Zhaoyu Liu, Chen Dong, Jia Wei Chen, Alvin Min Jun Jiang. Guanzhou Chen, Aayan Faraz Shaikh, Tian Yu Dong, Chen Wang, Kan Jiang, and Jin Song Dong. Analyzing the formation strategy in tennis doubles game. SN Computer Science, 6(2):100, 2025.
[49] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.068 06353, 2020.
[50] Hassan Mkhallati, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogen- broeck. Soccemet--capionn Dense video captioning for soccer broadcasts commentaries. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5073-5084, 2023.
[51] Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine-grained action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 122-132, 2020.
[52] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3163-3172 2021.
[53] Sangmin Oh, Anthony Hoogs, Amitha Perera, Naresh Cuntoor, Chia-Chih Chen, Jong Taek Lee, Saurajit Mukherjee, JK Aggarwal, Hyungtae Lee, Larry Davis, et al. A large-scale benchmark dataset for event recognition in surveillance video. In CVPR 2011, pp. 3153-3 3160. IEEE, 2011.
[54] OpenAL. Gpt-4 technical report, 2023.
[55] Luca Pappalardo, Paolo Cintia, Paolo Ferragina, Emanuele Massucco, Dino Pedreschi, and Fosca Giannotti. Playerank: data-driven performance evaluation and player ranking in soccer via a machine learning approach. ACM Transactions Intelligent Systems and Technology (TIST), 10(5):1-27, 2019.
[56] Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, and Yu- Wing Tai. Memory- attended recurrent network for video captioning. In The IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8347-8356. 2019.
[57] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. (0428-10436 2020.
[58] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for fine tne--gaiied action understanding. In The IEEE/CVF conference on computer vision and pattern recognition, pp. 2616-2625, 2020.
[59] Dingfeng Shi, Yujie Zhong, Qiong Cao, Lin Ma, Jia Li, and Dacheng Tao. Tridet: Temporal action detection with relative boundary modeling. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18857-18866, 2023.
[60] Zheng Shou, Dongang Wang, and Shih-Fu Chang. Temporal action localization in untrimmed videos via multi-stage cnns. In Proceedings of the IEÉE conference on computer vision and pattern recognition, pp. 1049-1058, 2016.
14

---

Published as a conference paper at ICLR 2025
[61] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
[62] Roman Voeikov, Nikolay Falaleev, and Ruslan Baikulov, Tinet: Real-time temporal and spatial video analysis of table tennis. In The IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops, pp. 884-885, 2020.
[63] Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In The IEEE conference computer vision and paltern recognition, pp. 7622-7631, 2018.
[64] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos. IEEE transactions on pattern analysis and machine intelligence, 41 41(11):2740-2755 2018.
[65] Wei-Yas Wang, Hong-Han Shuai, Kai-Shiang Chang, and Wen-Chih Peng. Shuttlenet: Position- aware fusion of rally progress and player styles for stroke forecasting in badminton. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4219 4227, 2022.
[66] Wei-Yas Wang. Yung-Chang Huang, Tsi-Ui Ik, and Wen-Chih Peng. Shuttleset: A human- annotated stroke-level singles dataset for badminton tactical analysis. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1126-136, 2023.
[67] Zhenzhi Wang, Ziteng Gao, Limin Wang, Zhifeng Li, and Gangshan Wu. Boundary-aware cascade networks for temporal action segmentation. Computer EsioonECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pp. 51. Springer, 2020.
[68] Dekun Wu, He Zhao, Xingee Bao, and Richard P Wildes, Sports video analysis on large-scale data. In European Conference Computer Vision, pp. 19-36. Springer, 2022.
[69] Jinglin Xu, Yongming Rao, Xumin Yu, Guangyi Chen, Jie Zhou, and Jiwen Lu. Finediving: fine-grained dataset for procedure-aware action quality assessment. In The IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2949-2958, 2022.
[70] Mengmeng Xu, Chen Zhao, David Rojas, Ali Thabet, and Bernard Ghanem. G-tad: Sub- Sub-graph localization for temporal action detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10156-10163, 2020.
[71] Fangqiu Yi, Hongyu Wen, and Tingting Jiang. Asformer: Transformer for action segmentation. arXiv preprint 11.00221,0 2021.
[72] Chen-Lin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers, In European Conference on Computer Vision, pp. 492-510. Springer, 2022.
15


Reviews:
[{"review_id": "jS3osOy2uk", "reviewer": "Reviewer_JNET", "summary": "Thanks the authors for this ICLR2025 submission.\n\nSummay:\n\nThis paper makes contribution to faciliating the detection of fast-speed, high frequence and fine-grained action events in videos. A new dataset, specific to **tennis**, is proposed to this purpose, together with an video frame-wise label annotation tool, which enables the collections of accurate labels. In experiments, a novel model is proposed and benchmarked on temporal action detection over 3 levels of granularity. Some ablation study is provided. To showcase the scalabiility and generalizability of proposed apporach, the paper extends the study to several other action-sports datasets (e.g., diving, gym, etc).", "questions": "see weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Writing is easy to follow.\n- The proposed dataset is collected with notable efforts, and is very-well organzied and presented. So does the 3-step annotation procedure, a very practical approach to the video dense annotation problem.\n- The appendix further provides comprehensive stats.\n- The experimental results are strong.", "weaknesses": "In general, there are a few minor weakness spotted in the experiment section which puts question marks to the technical sound of this paper.\n\n- (minor) The statement on line 398-399, `...it is crucial to utilize frame-wise feature extraction [7]`, is not well supported. The author might have gained much insights through some hidden experiments that clip-wise feature extration is inferior to frame-wise methods. Yet, it is less clear to the general audience. \n- (minor) It would be more insightful to provide the numeric impact of the **Event localizer** to the whole F^3ED system. As it would be a concern if a very good performing LCL module is the **hard-prerequisite**, in which case the generalization of the F^3ED approach is bit questionable. In some scenarios,  a good LCL might not be possible to have.\n- (minor) There is a lack of in-depth discussion why GRU based method is adopted in the CTX stage, given the transformer module has been demonstrated more efficient on long-range context modeling. \n- (minor) Figure 4 should be more clear on 1) what is the \"plus\" symbol that combines outputs of LCL and MLC, and 2) what are the red squres in feature vectores under the CTX module.\n\nMissing reference:\n- Sports video analysis on large-scale data, ECCV 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WR8th2LQa5", "reviewer": "Reviewer_bJ2T", "summary": "This paper mainly introduces a dataset for advancing fast, frequent and fine grained video processing tasks.\n\nThey also introduce a model built on top of the newly introduced dataset, and show improvements over several variants of existing video encoders and temporal fusion modules, eg, transformer-based modules, etc.\n\nThey choose some video encoders such as SlowFast, TSN and TSM, with different head architectures and compared to their proposed model. Experiments on 3 levels of granularity show the effectiveness of the model.\n\nThey also evaluate their module on top of TSM video encoder, on some fine grained benchmarks and show consistent improvements.", "questions": "Please refer to my concerns above.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "+ The efforts of collecting a dataset for fast, frequent and fine grained events are appreciated.\n\n+ The authors identify an interesting research problem, in terms of fast, frequent and fine grained video-related research.\n\n+ Table 1 is a nice and interesting comparisons.\n\n+ Some interesting comparisons and experiments conducted, and the authors also provide some valuable insights, especially section 5.1.", "weaknesses": "Major: \n\n- The dataset currently focuses on tennis swing, the contribution is limited (i) how your dataset differs from existing dataset and (ii) the dataset at this stage is only tennis, what would be the scope and focus of this benchmark.\n\n- The paper is written in a rush, and not well-structured. The authors mention that “our benchmark can contribute towards a VideoNet for future LLM’s benchmarking and finetuning”, but where and how? The contributions listed are also a bit vague and weak at this stage.\n\n- A review of existing closely related works for the concept of “fast, frequent and fine grained” is not performed; however, the paper introduces the new model (i) any existing works in the literature and how the proposed method differs from existing works (ii) any insights from reviewing existing works, eg, any practical concerns that highlight the importance of introducing a new model, etc.\n\n- Sec. 3 is not well-written and well-structured. The reviewer suggests the authors refine this work, and make it solid enough for next venue. Having a complete understanding, review and analysis in this work, would be much appreciated.\n\n- The justification of choosing evaluation metrics is not provided, and how these evaluation metrics contribute to the evaluation of performance etc. Section 5 page 7 bottom, what does it mean by “we have adapted these methods to develop …”, what methods and how this is performed?\n\n- Regarding evaluations, in Table 2, the use of video encoders is a bit too limited, although the reviewer likes the interesting comparisons presented. The authors should refine their research aims and make it better and clear. The current scope a bit too big, and also core evacuations a bit too limited. This would affect the impact of the work.\n\n- Showing the evaluation results in the form of list is not good, eg, in section 5.1. More detailed discussions are indeed needed to show more in depth analysis and comparisons. Although the paper discusses some insights, the paper at this stage is too raw and requires further efforts to make it solid and thorough enough. Last sentence of section 5.1 does not provide much information. “… achieves optimal performance among all methods”, how and why?\n\nMinor:\n\n- The fine grained concept should be first introduced in the introduction section, eg what is fine grained recognition tasks in videos, and what is the concept of frequent and fast.\n\n- All the experimental results and evaluations are presented in the form of tables. The authors are encouraged to use some plots, visualisations to make the comparisons clear and vivid to researchers and readers.\n\n- As a research paper, it is suggested to make it as clear as possible, eg “Conflicting samples were resolved using a majority vote criterion”, but how?\n\n- A notation section detailing the maths symbols used in the paper would be better.\n\n- Section 4, the proposed model is also unclear to reviewer. Inside problem formulation, what is that “j”? Fig. 4 is also not clear to reviewer. The figure caption does not provide much information to understand the figure.\n\n- In Introduction section, what is “T”, body and wide? These concepts are unclear to reviewer. The concepts are not very well-explained.\n\n- Incomplete sentence in sec. 3.1, “If it lands in bounds after crossing the net.”, it would be better to have a figure explain these concepts clearly in lexicon section would help readers. Fig. 3 suffers from low resolution, and the fonts are unable to read due to its too small nature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lPGxCPtD9W", "reviewer": "Reviewer_YGio", "summary": "The paper introduces **F3Set**, a novel sports-related dataset designed to address the challenges of detecting fast, frequent, and fine-grained (F3) events from videos. The dataset contains over *1,000* event types annotated with precise timestamps, primarily focused on tennis, but also includes other sports such as badminton and table tennis, with the potential to be extended to various other sports. To tackle the challenges of event classification and localization, the authors propose **F3ED**, a model that utilizes a video encoder for spatial-temporal feature extraction, a multi-label event classifier, and a contextual module to refine event sequence predictions. The model is evaluated on F3Set and demonstrates superior performance over certain models in both event localization and classification. Additionally, the authors provide a semi-automated toolchain for annotating F3 events, making the dataset scalable for use in other sports.", "questions": "- **Generalization of Methodology to Other Sports**: How do you expect the proposed methodology to generalize to sports with faster movements and multiple players, such as **soccer** or **basketball**? Can the model handle the requirements for higher frame rates and multi-human event interactions, or are there adjustments needed for such cases?\n\n- **Use of Low-Resolution Input (224x224)**: The decision to use **224x224 crops** despite collecting higher resolution videos raises questions. Have you tested the model on **higher resolution inputs**, and if so, how did it perform? Could higher resolution provide more visual details and improve the model’s ability to detect subtle distinctions?\n\n- **Alternative Approaches for Event Detection**: The paper focuses on comparing the proposed method with models using **similar architectural foundations** like 3D CNNs. Have you considered comparing your method to **fundamentally different approaches**, such as **pose estimation** for event detection (In other words: extracting poses and then utilizing coordinates to classify events)? How would such comparisons impact performance in terms of accuracy, speed, and interpretability?\n\n- **Dataset and Camera Perspectives**: The dataset appears to focus on controlled environments, but real-world scenarios often involve **different camera angles, lighting conditions, and varying court types**. How would your model perform under different camera perspectives or in less controlled environments? Have you tested the model with variations in lighting or weather conditions?\n\n- **Scalability of Annotation Process**: The paper mentions a **semi-automated toolchain** for annotating events. How scalable is this annotation process, especially if applied to larger datasets or more complex sports with multiple participants?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **Originality**: The paper introduces a dataset designed for detecting **fast, frequent, and fine-grained events** in sports videos. While there have been other tennis-related datasets, such as **Tennis Stroke Dataset** and **TennisDB**, **F3Set** stands out due to its **1,000+ annotated event types** and precise annotations with fine-grained details.\n\n- **Quality**: The dataset contains over **11,584 video clips** of professional tennis matches, featuring **42,846 tennis shots** and detailed annotations. Each shot is labeled with attributes like shot type, direction, and outcome, allowing for comprehensive event analysis. Also dataset has been collected in high resolution and moderate fps 25-30, which can be utilized for tasks beyond event classification, such as pose estimation and movement analysis.\n\n- **Clarity**: The authors provide clear examples of how annotations are done, such as distinguishing between forehand and backhand shots, and detailing shot outcomes like winners or errors. The explanations of **multi-level granularity** make it easy to understand how the dataset can be applied to different tasks, from coarse to fine event classification. In addition, the authors did a very good job providing the explanation of tennis terminology. \n\n\n- **Code and Tools**: The authors provide a **well-structured and anonymized codebase**, along with a semi-automated toolchain for efficient event annotation.  This makes it easier for researchers to adopt the dataset and extend it to other domains.\n\n- **Dataset Diversity**: The dataset includes **high-resolution videos** from 114 professional tennis matches featuring both men and women players, with frame rates of 25-30 FPS. This diversity, along with specific annotations for both right- and left-handed players, ensures the dataset can support a wide range of analyses.", "weaknesses": "- **Significance**: While the dataset is beneficial for sports like tennis, badminton, and table tennis, the methodology is highly limited due to the nature of these sports. These sports are relatively easier to model with their controlled environments and predictable movement patterns, but the methodology may not generalize well to faster and more complex sports like **soccer** or **basketball**, which require higher FPS rates and need to account for multi-player interactions.\n\n- **Benchmark Simplicity**: The proposed benchmark, while interesting and comprehensive in its approach, is relatively simple. The choice to crop the input videos to **224x224 resolution** while originally collecting them in higher resolution raises questions. The authors claim that F3ED outperforms more complex models like **SlowFast**, but this might be due to the limited resolution of the input images, which could fail to capture subtle visual distinctions. More work is needed to confirm or debunk this claim, including testing higher-resolution inputs to better understand the effects of image quality on model performance.\n\n\n- **Dataset Nature**: While the dataset is relatively diverse, there are still questions regarding the impact of **camera angles**, **court types**, **weather conditions**, and **illumination**. In real-world settings, these factors can vary significantly and may affect model performance. While in professional competitions these variables might be more consistent, in practical scenarios such variations could play an important role in the robustness of event detection. Ideally, dataset and benchmark should thoroughly address those concerns.\n\n- **Fit with ICLR and Representation Learning**: A notable weakness of this paper is that it does not explicitly address how the proposed model learns **representations** or how these representations could be generalized or transferred to other domains. Additionally, the paper primarily compares its performance to models with similar architectural structures, such as 3D CNNs, without exploring **fundamentally different approaches** to event detection. For instance, instead of relying on crops, the authors could have explored using **pose estimation techniques** to detect human poses and tackle the problem from a different perspective. Comparing such an approach across metrics like accuracy and speed, and then resonating on why one method outperforms or underperforms the other, would have provided valuable insights into the advantages or limitations of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)