EAST: Early Action Prediction Sampling Strategy with Token Masking
Authors: Iva Sović
Ivan Martinović
Marin Oršić
Venue: ICLR 2026 Poster (2026)
Keywords: early action prediction
token masking
video analysis
efficient training
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Early action prediction seeks to anticipate an action before it fully unfolds, but limited visual evidence makes this task especially challenging. We introduce EAST, a simple and efficient framework that enables a model to reason about incomplete observations. In our empirical study, we identify key components when training early action prediction models. Our key contribution is a randomized training strategy that samples a time step separating observed and unobserved video frames, enabling a single model to generalize seamlessly across all test-time observation ratios. We further show that joint learning on both observed and future (oracle) representations significantly boosts performance, even allowing an encoder-only model to excel. To improve scalability, we propose a token masking procedure that cuts memory usage in half and accelerates training by 2× with no accuracy loss. Combined with a forecasting decoder, EAST sets a new state of the art on NTU60, SSv2, and UCF101, surpassing previous best work by 10.1, 7.7, and 3.9 percentage points, respectively.

EAST: EARLY ACTION PREDICTION SAMPLING STRATEGY WITH TOKEN MASKING
Iva Sovié, Ivan Martinovié, Marin Oršić
Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia (iva. sovic, c, ivan. marti martinovic, marin. marin.orsic)eeer. orsic)
ABSTRACT
Early action prediction seeks to anticipate an action before it fully unfolds, but lim- ited visual evidence makes this task especially challenging. We introduce EAST, simple and efficient framework that enables model to reason about incomplete observations. In our empirical study, we identify key components when train- ing early action prediction models. Our key contribution is randomized train- ing strategy that samples time step separating observed and unobserved video frames, enabling single model to generalize seamlessly across all test-time ob- servation ratios. We further show that joint learning on both observed and fu- ture (oracle) representations significantly boosts performance, even allowing an encoder-only model to excel. To improve scalability, we propose a token mask- ing procedure that cuts memory usage half and accelerates training by with negligible accuracy loss. Combined with forecasting decoder, EAST sets a new state of the art on NTU60, SSv2, and UCF101, surpassing previous best work by 10.1, 7.7, and 3.9 percentage points, respectively.
INTRODUCTION
Action recognition enables machines to identify, understand and interpret human activities in video (Bobick & Davis 20011 Karpathy et al., 2014). Many important applications of this task require hard real-time inference in order to ensure timely reaction or precautionary measure. Ex- amples include security surveillance (Wren et al. 1997). human-robot interaction (Breazeal 2003). autonomous driving (Geiger et al. 2012). workplace safety, and other safety-critical applications. Such applications benefit from accurate predictions even before the action took place in its entirety. This state of affairs motivates subtask known early action prediction or early action recogni-
tion (Hu et al. 2019 Foo et al., 2022 Kong et al. 2017 Stergiou & Damen 2023
Ryoo 2011).
Early action recognition methods classify actions from partially observed part of the video (Ryoo 2011]. This makes the task challenging since the model should consider upcoming future content that is inherently multi-modal distribution (Vondrick et al. 2016 Baltrušaitii al.. 2019). Recent
methods find future action cues using auxiliary methods that do not always benefit early action classification performance, such motion forecasting (Pang et al. 2019 Liu et al. 2023). future residual forecasting (Zhao & Wildes 2019) or modelling the possible future state using graphs (Wu al. 2021b). Furthermore, the latest methods require separate models for each observation ratio. This requires immense training resources and complicates model deployment.
In this work, we propose EAST (Early Action prediction Sampling strategy with Token masking), an end-to-end framework that learns to predict actions from partial observations more effectively and efficiently. The core concept within EAST is a frame sampling strategy that enables training a single model for all observation ratios. During training, EAST samples partially observed (present) videos for all observation ratios, as well as full videos (future). Compared to methods that train per observation ratio models, our strategy simplifies inference and speeds up training 9x when there are 9 observation ratios. In contrast to previous methods that use auxiliary objectives, we simplify greatly
the learning objective by directly optimizing action prediction performance. Moreover, improve training efficiency by masking input patches that change the least over time. Remarkably, find that much as 50% of tokens can be removed without significantly degrading performance.

---

Token masking reduces inference time, but primarily aims efficient training: total GPU time and memory reduces by 2x, allowing EAST to train using two GPUs with 20GB of memory.
EAST involves three main contributions. First, propose a framework that trains single model based on classifications of common encoder features from dynamically sampled present and fu- ture video frames. We achieve further improvements using a forecasting decoder over present features. The proposed setup greatly improves efficiency since a single model is tested across all observation ratios. Second, we improve training efficiency by removing repetitive tokens ac- cording to visual similarity of input patches. Third, we evaluate our contributions through ex- tensive validations on standard action classification datasets. EAST sets the new tfat-oof-the-at across all evaluation settings for early action prediction on NTU60 (Liu et al. 2019), Something- Something V2 (Goyal et al., 2017 and UCF101 (Soomro et al. 2012). Source code is available at https: : / thub. com www.oviccvvisstt...o
2 RELATED WORK
Action recognition strives to interpret human activities after observing the entire video. The sem- inal approach by Karpathy et al. (2014) finds temporal structure by combining independent 2D convolutions, while |Simonyan & Zisserman (2014) propose separate appearance and motion pro- cessing. Spatio-temporal features naturally extracted with 3D convolutions (Ji et al. 2012 Tran (ct al., 2015; Lin et al. 2019 Feichtenhofer et al.) 2019). These models benefit from ImageNet
by repeating pre-trained 2D convolutional kernels into the temporal dimension (Deng et al. 2009
Carreira & Zisserman 2017). However, convolutional architectures struggle with long-term spatio-
temporal features and excessive model complexity (Wang et al.) 2016: Feichtenhofer et al.) 2019
Xie et al. 2018 Tran et al. 2015). Therefore, the most recent work favours transformer-based ap-
proaches (Piergiovanni et al. [2023] Li al. 20233 |yyai al. 2023
Li et al. 2022c
Tong al.,
2022
Wang et al. 2023] Srivastava & Sharma 2024)
VIT token removal. Masked image modelling is an effective self-supervised pretext task (Doso- vitskiy et al. 2020; He et al. [0222] Zhou et al. [022]] Gupta et al. (023). Fortunately, masking input tokens greatly reduces training time and memory complexity. This is especially important for long videos due to quadratic complexity of self-attention. VideoMAE and MAE-ST extend masked image modelling to video using a very high masking ratio of spatio-temporal cubes known as tubelets (Tong et al. 2022 Feichtenhofer et al., 2022: Piergiovanni et al. 2023 He et al. 2022 VideoMAE V2 extends the masking to the image reconstruction decoder (Wang et al. 2023).
Token masking also benefits supervised training. NaViT trains on combinations of entire and sub- sampled image tokens (Dehghani et al. 2023). Dynamic hierarchically prunes redundant tokens
in online manner (Rao et al. (2221) EVEREST removes tokens from uninformative frames and redundant patches (Hwang et al. 2024). Other approaches reduce tokens by similarity (Liang et al,
2022
Bolya et al.] 2023 Fayyaz et al. 2022 Yin et al. [222]] Haurum et al. [023] Choudhury al.
2024
DTEMM decouples feature representation learning from token merging (Lee & Hong 2024
Our token masking optimizes training while retaining early action prediction accuracy.
Action anticipation methods predict future actions before they hegin. A prominent approach an- ticipates future frames in unlabeled video (Vondrick et al. 2016). AVT anticipates actions with per-frame encoder and supervised causal decoder (Girdhar& & Grauman 2021). Fernando & Herath (22TT) supervise feature forecasting using Jaccard similarity between observed and future Teatures. Our approach does not optimize a feature similarity measure. Instead, we use video-specific en- coder and include a novel discriminative loss the training objective.
Early action prediction considers methods that classify partially observed videos (Wang et al., 2019). Early work considers action dynamics from a bag of visual words (Ryool 2011). or us- ing LSTM memory that records early observations (Hochreiter & Schmidhuber] 1997] Kong et al.)
201 Sah. ERA finds subtle early differences between actions with a mixture of experts (Foo et al.,
2022 Jacobs et al. 1991). IGGNN+LSTGN models spatio-temporal object relationships using fea- tures around bounding box detections (Wu et al., 2021a). MSRNN uses soft regression on early frame features account for future action uncertainty (Hu et al. 2019). Early-ViT learns action prototypes to regularize partial video representations (Camporese et al. (023). TemPr processes a
temporal feature cascade using consensus between transformer towers (Stergiou & Damen 2023).
2

---

E
VIDEO ENCODER
DECODER D
mean Foot
Good
Cree
Lime
shored porometers
shared porometera
E
VIDEO ENCODER
mea poot
comele
Figure 1: EAST both present and future frames in training. VIT encoder & processes the observed frames (blue) and the entire video (blue and red). Decoder 'D observes present features E. and forecasts future features F. h classifies actions from decoder features and oracle encoder features E. We optimize both classification scores but only yprd during inference.
Similar us, some early action prediction methods guide anticipative future representations by training with entire videos. DBDNet trains a Bi-LSTM that bidirectionally reconstructs present and future motion (Pang et al. 2019). LST-GCN models
evolution of object relation- ships using graph convolutional networks (Wu et al. 20216). AA-GAN forecasts future representa- tions by leveraging optical flow (Gammulle al. 2019). Furthermore, AA-GAN enhances future representations using adversarial training, where a discriminator discerns between generated and or- acle future features. Similarly, an action recognition teacher can supervise the student that receives only early video frames (Wang et al. 2019). DeepSCN starts by learning enriched features that min- imize the discrepancy between partial observations and full observations (Kong et al. 2017). Con- sequently, it learns SVM model to classify enriched partial features into categorical actions. iZhao & Wildes (2019) propose to forecast the future residuals with Kalman filter and then recursively in- tegrate them into feature representations of unobserved frames that separately classified. Unlike all previous approaches, we express the recognition of partially forecasted and completely observed sequences purely using discriminative losses. The classifier infers from pre-trained encoder features and also from forecasted decoder features within end-to-end training. Most importantly, our training strategy samples observation ratios when preparing training samples. This procedure enables good generalization with an arbitrary observation ratio.
METHOD
Early action prediction involves making predictions while observing a fraction of the video. There are Ta video frames. The observation ratio p (0, controls the fraction of observed (present) frames. Therefore, the model predicts early actions based on frames [0, p Ta). In training, the model has access all Ta frames and one-hot annotations y. In inference, the model makes predictions based on the first p Ta frames, where standard practice evaluates using from 0.1 to 0.9 in increments of 0.1. We follow this setting and apply unified model all observation ratios.
There are three main parts in EAST. First, our frame sampling strategy enables training a single model all observation ratios by sampling observed and unobserved clips. Second, optimize an objective that enforces correct predictions from observed frames and also from entire clips. Third, reduce the video transformer training memory using token masking based visual repetitive- ness, without compromising accuracy. Next, we explain the details of these steps, beginning with the most important: frame sampling.
3.1 SAMPLING STRATEGY FOR TRAINING EARLY ACTION PREDICTION
In training, we randomly sample p E (0.1, 0.2, 0.3, 0.9}. Using p. we collect T' observed frames V E BT and 7 unobserved frames v Rrxuaaxe so that Vo temporally precedes Ta and V succeeds p- Ta. The sampled clip V= V consists of 27 evenly spaced frames. We ensure that the final frame in V and the first frame in V are adjacent frames in the original video, avoiding temporal distortion training samples. Randomizing p in training enables the model to adapt to variable temporal context length.
3

---

Although conceptually simple, this training setup is essential for early action prediction. Our prelim- inary experiments suggest that training at fixed observation ratios produces models that are subopti- mal at other values of p. Such setup would require training specialized models and hinder real- world applications. Furthermore, we find that off-the-shelf action recognition models fail on early action prediction since they depend the full context.
3.2 FORECASTING WITH MAE FEATURES
The encoder architecture closely follows Vision Transformers (ViT) with ppatio-temport positional encodings to account for video (Vaswani et al. 2017, Dosovitskiy et al., 2020). The
Encoder.
ViT-based encoder & consists of tokenizer 'I and transtormer encoder V. Concretely:
E. +R NexF.
(Vo
(1)
The model processes input frames with C 3 RGB channels. The encoder extracts tokens with F' features. Tokenizer T splits the input clip frames into N, THW non-overlapping tubelets of
y'd size d p.p, where p= 16 and d=2 determine the spatial and temporal tubelet size, respectively. Spatio-temporal information is added to tokens via sin-cos embeddings. The transformer encoder NIXF Nexe extracts features from the input video clip.
Decoder D forecasts future features F given E
D:R +R
"OE")=FoPPEE'-E
The encoded present features E" EV") are input spatial average pooling PR:RNer +R TxP that produces mean token for each time step. Decoder module JF processes Σ present tokens and forecasts I future tokens.
We produce a strong baseline by setting F to an identity mapping, effectively making the method decoder-free. We further evaluate the decoder design with two distinct architectures: i) autoregres- sive and ii) direct transformer. Autoregressive formulation of F observes E" and forecasts tokens with causal inference. Direct inference concatenates E with additional [MASK] tokens, and per- forms single forward pass through a full attention transformer. Based on the validation results, we set the direct 4-layer transformer as within EAST.
3.3 COMPOUND FORECASTING LOSS
Figure contains diagram of the training setup. The partially observed clip V is classified using:
y h P: D EVV).
(3)
Here, h produces early action classification logits using linear layer, and P. denotes pooling.
The encoder features should be both discriminative and contain cues about future features. To achieve this in training, we perform additional forward pass through E. We compute oracle encoder features using the entire sampled clip: E = P. EVV). Consequently, the common classi- fier produces two sets of classification logits. The first set geree = h o P. (F) contains early action classification logits obtained by forecasting from E whereas the second vector gorack: h P,(E) contains classification logits for the entire sampled video clip.
We train the model from end-to-end minimize the average compound loss that sums negative log-likelihoods:
L + Coracle = ENLL (y .y) + LNLL yy)
(4)
This formulation is intuitive when considering the loss gradients. Gradients through grdd directly optimize early action prediction and enforce discriminative features in both the encoder and the decoder. Gradients through coaale yield discriminative features when observing a full video. We find that the combination of the two losses yields the best early action prediction.
3.4 EFFICIENT TOKEN MASKING
To reduce computational costs of attention layers, we propose to mask temporally repeating tokens. The proposed token masking strategy has been inspired by the Moravec corner detector (Harris)
4

---

& Stephens 1988). This masking strategy primarily reduces the training memory footprint, mak- ing EAST suitable for training on more affordable GPU setups. Also, the proposed masking reduces inference time by reducing the number of input tokens.
We find repeating tokens according to LI patch distances throughout time (Choudhury et al. 2024). Tubelet volume is set by patch size p and tubelet size d. Thus, extract vectorized non-overlapping patches Pr,ij using:
Pr..
(5)
In each frame from video V. we rank each tubelet according to pixel distance from the last patch in the next tubelet:
Te.s. (V) = [t. - Pt++,,5(4-1jll
(6)
Note that we compare the first and the last patch since size > 1. Finally, we keep the highest ranking tubelets using:
(V) {Pr.ss ( 2 r/.
r denotes ranking for the k-th quantile spatial position (i, We set A: = 50% in our ex- periments and apply token masking when computing both present and oracle features. Note that masking with & removes the same number of tokens from each spatial position. In other words, halve the number of input tubelets at each spatial position. We refer to M as difference masking. In training, we apply M independently to V and V prevent information leakage. Note that we use feature extractors pre-trained with MAE (He et al., 2022 Tong et al. 2022). Therefore, the encoder unaffected by masking since there is no distribution shift compared to MAE pre-training.
4 EXPERIMENTS
We compare EAST with related methods on four datasets used in the early action prediction setup: Something- Sanething-Something.. versions v2 and sub21 (Goyal et al. 2017), NTU RGB+D (Liu al. 2019). UCF101 (Soomro et al. 2012) and EK-100 (Damen et al. 2022). We also include experiments that measure the influence proposed components. Refer to the supplement for detailed insights.
4.1 DATASETS
Something-SSomthhine v2 (SSv2) is large-scale video dataset primarily used for action recogni- tion. The dataset consists of 220 k video samples and 174 classes. There are 169 k training videos and 20 validation videos, whereas the remaining unlabeled videos are used for testing. SSsub21 is a Something Somiiting subset typically used in early action prediction evaluation. It contains 21 action classes across 11 videos. We include experiments on SSsub21 to compare with most previ- ous methods. We also include results on the full SSv2 dataset.
NTU RGB+D dataset consists of 60 action classes and has 57k 1920x1080 RGB videos. Most samples also include depth maps, infrared frames and skeletal keypoints. We use only the RGB modality to train EAST. Following previous work, we use cross-subject evaluation our experi- ments (Ma et al.] 2016 Kong et al. 2017 Hu et al.) 2019 Wang et al. 2019 Pang et al. 2019 Stergiou & Damen 2023). There are 20 subjects in both training and evaluation sets. This split provides 40.3 training examples and 16.5 k testing examples.
UCF101 is a small scale dataset that consists of approximately 13 k videos with 101 action classes. Videos are divided into 9.5 k training and 3.5 k validation videos. The video resolution is 320x with a frame rate 25 FPS.
Epic- Soic-Kitchens-100 (EK-100) is large collection of egocentric videos consisting of 90 thousand action segments. The taxonomy consists of verb and nown category groups that determine the action group. The evaluation considers classification in all three category groups
4.2 IMPLEMENTATION AND TRAINING DETAILS
Unless otherwise stated, we the VIT-B/16 video encoder pre-trained on K400 using Video- MAE (Kay et al. 2017; Tong et al. 2022). Decoder F is also initialized from VidcoMAE pre- training. This yields slight improvements random init. We train the entire model end-to-end.
5

---

We sample T 8 frames for both V and V", and train random 224x224 crops using MixUp augmentations (Zhang et al. 2018). We use AdamW W with base learning rate x 10 and weight de- cay 0.05. The base learning rate scaled by hatch size and decayed using the cosine rule (Loshchilov & Hutter 2016). We set the batch size to 96 in SSv2, NTU60 and EK-100 experiments. In SSSub2T and UCF101 experiments,
25
set the batch size 128. We train SSv2 and EK-100 for 40 epochs, and on NTU60 and UCFI01 for 50 and 100 epochs, respectively. We report results from a single training run that uses fixed random seed. We express the computational complexity of forward pass over single training example. We measure this complexity as the number of floating point operations (TFLOP) using DeepSpeed (Rasley al., 2020). We train with MixUp, therefore, our measurements reflect the complexity of processing both augmentations.
We use Nvidia RTX A6000 GPUs. All our experiments use FlashAttention optimizations that saves memory by recomputing the attention matrix in backpropagation (Dao et al. 2022). In addition to the memory savings from FlashAttention, applying M further reduces training memory by x. The proposed training requires only 2 x A6000 GPUs. Unless otherwise stated, we set 0 and perform the same masking in training and inference.
Since most previous work did not publish source code, evaluation details are not fully disclosed. We propose a unified protocol for early action prediction via minimal adaptations of action recognition evaluation. We apply single model across all nine observation ratios and report top-1 accuracy. The model is agnostic to the testing observation ratio and processes present frames only. We do not subsample features pre-computed from entire videos since that would lead to unfair leak of information. We perform spatial multi-crop inference (Feichtenhofer et al. (2019). On NTU60 and SSv2, we average predictions when sliding across temporal dimension (Wang 2016
4.3 COMPARISON WITH THE STATE OF THE ART
NTU60. Table D shows results on the NTU60 dataset. The first section includes methods that process multiple modalities (skeletal keypoints or depth). The second section presents methods that only use RGB input frames. EAST surpasses all methods while using only RGB inputs. The average improvement over TemPr is 6.8 pp, with the highest improvement of 19.2 pp at p=0.3.
Table 1: Comparison with previous work on the NTU60 dataset. We show top-1 accuracy (%) over different observation ratios. denotes reproductions by Wang al. (2019). We highlight input modalities as video (RGB), depth (D) and human keypoints (KP). The best results are in bold.
Modality D
Ohservation ratio
Method
RGB
KP
0.1
0.3
0.5
0.7
0.9
MSRNN (Sadegh Aliakbarian et al.
2017)
15.2
29.5
51.6
63.9
68.9
TS (Wang et al. 2019)
/
27.8
46.3
67.4
77.6
81.5
DBDNet (Pang 2019
/
/
28.0
47.3
68.5
78.5
81.6
STM* (Ma al.
2016
11.5
25.7
48.0
61.0
66.1
DeepSCN (Kong
16.8
30.6
48.8
58.2
60.0
TemPr (Stergiou Damen 2023
29.3
50.2
70.1
78.8
84.2
EAST
31.2
69.4
86.2
87.9
87.9
Table 2: Comparison with the state-of-the-art results SSv2 dataset. We show top-1 accuracy (%)
over different observation ratios. The best results are in bold.
Observation ratio p
TFLOP
Method
0.1
0.2
0.3
0.4
0.5 0.6 0.7
0.8
0.9
RACK (Liu et al. 2023)
11.9
15.0
23.0
Early-VIT (Camporese el al.
2023
22.7
27.8
33.6
40.5
48.0
53.9
58.5 61.5 63.0
TemPr (Stergiou & Damen 2023
20.5
28.6
41.2
47.1
0.5
EAST
25.6 30.1 34,5 41.6 49.0 55.2 59.4 63.0 64.0
0.5
SSv2. Table presents our results on the Something- Something v2 dataset, where EAST sets the new state of the art. Average result improvement over TemPr is 28.3 pp. For observation ratios

---

Table Comparison with the state-of-the-art results on SSsub21.
We present top-1 (%) accuracy.
refers to results that are presented by Stergiou & Damen 2023).
The best results are in bold.
Observation ratio
Method
0.1
0.2
0.3
0.5
0.7
0.9
MS-LSTM (Sdegh Aliakbarian et al.
2017
16.9
16.6
16.8
16.7
16.9
17.1
MSRNN# (Sadegh Aliakbarian et al.
2017)
20.1
20.5
21.1
22.5
24.0
27.1
mem-LSTM IKong
2018a)
14.9
17.2
18.1
20.4
23.2
24.5
GGN (Wu al. 20216)
21.2
21.5
23.3
27.7
30.2
30.6
IGGN [wu et al. (021Ta)
22.6
25.0
28.3
32.2
34.1
TemPr (Stergoou & Damen
2023
28.4
34.8
37.9
41.3
45.8
48.6
Early- ViT Camporese et al
2023
32.1
35.5
40.3
52.4
60.7
62.2
EAST
40.8
44.7
51.2
66.4
75.8
79.3
Table 4: Comparison with the state-of-the-ard results on the UCF101 dataset. We show top-1 ac- curacies (%) over different observation ratios. denotes results reproduced by Kong et al.) (2017). TemPr entry with presents original paper results that irreproducible using the published source code. å represents our corrected reproduction.
Observation ratio p
Method
0.1
0.2
0.4
0.5
0.7
0.8
0.9
MSSC (Cao et al.
2013)
53.8
58.3
62.6
61.9
64.3
62.7
M (Kong et al.
2014
40.1
72.8
80.0
82.2
82.4
83.2
83.4
83.6
83.7
DeepSCN Kong ct al.
20177
45.0
77.7
83.0
85.4
85.8
86.7
87.1
87.4
87.5
MSRNN (Sadegh Aliakbarian
2017
68.0
87.2
88.2
88.8
89.2
89.7
89.9
90.3
90.4
mem-LSTM [Kong
20188
51.0
81.0
85.7
85.8
88.4
88.6
89.1
89.4
89.7
AAPNET Kong et al.
20188)
59.9
80.4
86.8
86.9
88.3
88.3
89.9
90.9
RGN-KF Zhao & Wildes
2019
83.3
85.2
87.8
90.6
91.5
92.3
92.0
93.0
92.9
DBDNet Pang et al.
2019
82.7
86.6
88.4
89.7
90.6
91.1
91.7
91.9
92.0
AA-GAN (Gammull et al.
2019)
84.2
85.6
(Wang
(0199)
87.1
88.9
90.9
91.0
91.2
GGN wu ct al. 202
84.1
88.5
89.8
90.9
91.4
91.8
IGGN (Wu ct al.
2021a
80.2
89.8
92.9
94.1
94.4
JVS Fernando & Herath
2021
91.7
ERA (oo
2022
89.1
92.4
94.2
95.5
95.7
RACK (Liu
al. 2023)
87.6
87.6
89.4
Early- VII Camporese el al.
2023
MoViNet
87.2
90.1
91.7
92.2
92.9
93.4
93.6
93.5
93.5
TempPr!
Stergiou Damen
2023
MoViNer
88.6
93.5
94.9
94.9
95.4
95.2
95.3
96.6
96.2
TempPr
Stergiou & Damen
2023
MoViNar
85.1
90.4
92.5
92.8
93.2
EAST videMAAE
88.6
91.4
92.2
93.1
93.4
93.6
93.7
93.8
93.8
EASTMaNNN
91.3
93.2
93.8
94.7
95.5
96.1
96.4
96.5
96.5
p= 0.1, 0.3, 0.5 and p= 0.7 we improve the results by 5.1 pp, 5.9 7.8 pp and 12.3 pp. respectively. Unlike TemPr, we train all parameters end-to-end while achieving similar TFLOP complexity, Furthermore, the results highlight the benefits of our proposed sampling strategy. Note that we evaluate single model whereas TemPr trains special model for each observation ratio. Therefore, TemPr requires the observation ratio during model inference, while our model entirely agnostic to the observation ratio. Finally, Table 3 presents results on SSsub21, where the average improvement over TemPr is 22.7 pp across all observation ratios.
UCF101. We present our UCF101 results in Table 4 Since TemPr uses a MoViNet (Kon- dratyuk et al., 2021) encoder pretrained on K600 (Carreira et al. 2018), we include results with the same backbone. These results highlight the benefits of training under our proposed framework. EAST with Mo sets the new state-of-the. of-the-an art results for observation ratios on UCF101. Av- erage result improvements over ERA and TemPr are 1.3 pp and 3.9 pp. respectively. These results show that our method does not depend on the backbone, and that improvements come from the proposed training strategy.

---

EK-100. We train and evaluate EAST on EK-100 using two classification heads, one for verb and one for noun. We use evaluation scripts provided by the official EK-100 repository. The results presented in Table 5 EAST outperforms TemPr low observation ratios but is limited by the ViT encoder accuracy at high observation ratios. TemPr uses the SlowFast hackbone that achieves top-1 action recognition accuracy of 38.5 5 %, 50.0 % and 65.6 % on all action, all noun and all verb, respectively. This outperforms VideoMAE ViT-B that achieves 33.7 %. 42.7 7 % and 65.1 % top-1 accuracy on all action, all noun and all verb, respectively. The results strengthen our contributions since EAST significantly outperforms TemPr on fow-observation ratios.
Table 5: Top-1 early action prediction accuracy on EK-100 under different observation ratios
All Action
All Noun
All Verb
P 0.1
0,2 0.3 0.5 0.7
0.1 0.2 0.3 0.5 0.7
0.1 0.2 0.3 0.5 0.7
Tempr 7.4 9.8 15.4 28,9 37.3
22.8 25.5
32.3 43,4 49.2
21.4 22.5
34.6 54.2 63,8
EAST 20.4 23.3 25.4
28.1 29.7
31.1 34.0 35.5 38.2 39.9
47.2 52.1 55.0 58.4 59.5
Results on all four datasets showcase that EAST generalizes in both small and large-scale datasets. Most importantly, we train one model for all observation ratios. We found that training separate models for each does not yield any accuracy improvements but requires 9x more training time.
4.4 ABLATIONS AND ANALYSES
We validate EAST on SSv2, unless otherwise specified.
Overlooked baseline EAST Table 6 presents early action classification performance trained on entire action classification sequences.
when the VideoMAE ViT-B/16 model EAST trains the same model but uses our proposed sampling. Unlike EAST, EAST; trains using Lode only and does not have a decoder (F is identity). When comparing EASTş to VideoMAE, there is a noticeable difference for smaller ob- servation ratios. For p 0.1, Video- MAE achieves only 9.9 % accuracy com- pared an encoder-only EAST& which achieves 23.9 %. Accuracy differences are higher observation ra- tios. This is expected since high observa- tion ratios include more context. Never- theless, the results indicate the critical impact of the appropriate sampling strategy. We establish new early action prediction baseline that clearly outperforms the current state of the art on SSv2, achieving an average improvement of 5.4
Table 6: Top-1 SSv2 accuracy over all observa- tion ratios. VideoMAE denotes pre-trained ViT-B/16 model performance finetuned for action classification. EAST trains ViT-B/16 with our proposed sampling without decoder.
Observation ratio p
Method
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
VideoMAE
9.9 14.8 20.3 29.4 39.5 49.2 52.2 61.5 63.4
less prominent
EAST: 23.9 28.3 32.7 39.1 46.1 56.0 56.5 59.6 60.7
Token masking and computational efficiency. Table validates our token masking method M on NTU60. We compare M to random masking Mirand from VideoMAE and Running Cell masking MMAR from MAR QQing al. 2023). We measure performance using three different masking ratios k E (25%, 50%, 75% The proposed difference masking outperforms other masking across tested masking ratios. This highlights the importance of retaining patches that contain most appearance change. We measure peak training memory (GB) for batch size 24 and count TFLOPs for a forward pass given one training example. As expected, masking k 0.75 patches most efficient, but it does not achieve state-of-the the-trr-ttt-tttto art results. Although M 25 model achieves highest accuracy, choose M 0.5 since this setup offers best balance between efficiency and performance.
Encoder-only vs encoder-decoder. The first row in Table g shows the accuracy of an encoder-only baseline. This corresponds to the EAST: entry from Table 6 The second row in Table 8 shows that training encoder-only EAST. using both gred and aracl gains additional 1.5 1.5pp. pp. An encoder- decoder model trained using ged and naalle yields EAST, improving the average accuracy by 0.6 pp over EAST. Training without Comale decreases results in both encoder-only and encoder-decoder setups. The results highlight the benefits of training using the proposed compound loss in both cases.
8

---

Table 7: Average NTU60 top-1 accuracy using difference masking M Running Cell masking MMAR and random masking M k denotes the percentage masked tokens. We report training complexity with example and peak training memory with batch size 24.
mode
EAST
0
-.. 0.8
50
accurary
40
Masking
avg, acc.
TFLOP peak mem. (GB)
64.0
M
M40.70
66.3
+2.3
0.2
10.4
M50.77
71.9
+7.9
M rand
71.3
Figure Comparison between EAST and mod- els trained for a single observation ratio on SSv2. Each line denotes accuracy differ- ent model at each observation ratio. Training single model with EAST comes near the accu- racy of models trained for particular p. Note that specialized models fail at observation ratios they were trained for.
M
72.4
+1.1
0.5
19.2
M
74.3
+3.0
M 25
73,4
M 0..2
MAR
73.3
-0.1
0.8
27.9
M 0.25
75.2
+1.8
no mask 75.1
1.1
36.7
Table 8: Contributions of the proposed losses and modules to SSv2 validation accuracy. D denotes the choice of the decoder. id denotes model where decoder 'D set to the identity mapping. Our encoder-only approach already surpasses the previous state-of-the-arr. Joint L and Loraele optimization benefits both encoder-only and encoder-decoder models.
Conacl goed
D
p=
p=0.2
0.2
p= 0.3
p=
0.4
p=
0.5
p= 0.6
p= 0.7
p= 0.8
p= 0.9
avg
id
23.9
28.3
32.7
39.1
46.1
56.0
56.5
59.6
60.7
44.8
id
25.3
29.4
33.6
40.3
48.0
54.5
59.2
62.6
63.9
46.3
/ /
26.1
30.4
34.5
41.2
48.3
54.1
58.2
61.0
61.8
46.2
/ /
/
25.6
30.1
34.5
41.6
49.0
55.2
59.4
63.0
64.0
46.9
Choice of the decoder. AVT (Girdhar Grauman 2021 suggests that autoregressive prediction natural in modelling temporal action progression for action anticipation. However, findings in Table 9 a) show that forecasting by direct decoding outperforms autoregressive decoding by an average of 0.6 pp. Both approaches are viable since they surpass the current best method. Due to slightly better results, we chose direct forecasting in EAST.
L2 loss. Alignment between observed and oracle features natural choice in guiding anticipative behaviour. However, Table b) shows that adding an L2 loss between oracle and predicted features lowers accuracy by an average of 0.3 pp. We noticed that the L2 loss minimizes the start of training. Our hypothesis is that strong feature alignment neglects some important temporal patterns.
Shared separate classifiers. Table ec) shows the contribution of the shared classification head. The shared classifier slightly outperforms two independent classifiers by average of 0.3 pp. with a maximum improvement of 0.7 pp at = 0.5. Parameter sharing enforces a consistent decision boundary between observed and predicted features. It also reduces the overall model complexity.
Wall clock time. We compare the time duration of one training epoch between EAST and TemPr. We use the same 4 A6000 server, and turn off unnecessary CPU-GPU communication. such loss logging. Mean measurements on UCF-101 are 80 seconds for EAST and 173 seconds for TemPr. EAST without masking trains epoch in 180 seconds, highlighting the token masking efficiency. During inference, EAST processes a video clip in 12 ms, whereas TemPr executes in 78 ms.
One per P model. Figure compares EAST with 9 models that specialize in single obser- vation ratio. Although specialized models mostly perform better at their respective training time observation ratio, they fail in most other setups. The results indicate that training with EAST yields

---

Table Validation of the decoder, loss choice, and classification head on SSv2 against top-1 accu- racy across different
a) Direct decoder Dair consis-
b) The inclusion of an L2 loss c) The shared classifer slightly outperforms separate classification heads.
tenly outperforms autoregres- sive decoder Dar.
in EAST yields further per- formance gains.
Observation ratio p
Observation ratio p
Observation ratio p
D
0.1
0.3 0.5 0.7
C2 0.1 0.3 0.5 0.7
cts
0.1 0.3 0.5 0.7
Dar
25.0
34.2
48.2
58.8
25.6
34.5
49.0
59.4
shared
25.6 34.5
49.0
59.4
Dai
25.6 34.5
49.0 59.4
v
25.7 34.4 48.2 59.1
separated 25.7
34.3 48.3 59.2
sufficient capacity to learn discriminative cues across different observation ratios, which plays crucial role in improving model performance. See supplement for more insights.
Qualitative examples. Figure B shows qualitative comparison between VideoMAE and EAST VIT- models. We show model outputs at different observation ratios p {0.1, 0.3, 0.5, 0.7, 0.9}. The examples show that VideoMAE outputs incorrect classification at small p. but its outputs are correct higher p. In contrast, EAST demonstrates the ability to identify the correct class starting from the lowest observation ratio 0.1. The examples highlight EAST's ability to extract discriminative cues given portion the action. More examples can seen the Appendix, Figure
0.1
0.3
0.5
0.7
0.9
VMMShowinng somsething top of something
VM-Droopping soanchingg
VM: Covering sometbing
VM: Covering something with somnehhing
VM: something
nest wonthing
with something
with something
EAST: Covering
some-
FAST: Covering
SOIRE
RAST: Covering
SOme-
FAST: Covering
some
EAST: Covering
NOmTE>
thing with something
thing with something
thing with something
thing with something
(hing with something
0.1
0.3
0.5
0.7
0.9
Showing something
Showing that something
Laning something
VM: Lilling
nomeching
VM: T.ahing
sumething
something
something
sumething chingaann
with something
with somothing
Lifting somathhig
Lifting
something
Lifting somethhin
EAST: Lifting something
RANT: Lifting
something
something
something
something on
Figure 3: Examples from the SSv2 dataset. We show the last frame within observation ratios. At the specified observation ratio X and / denote false and true model predictions, respectively. EAST makes accurate early predictions, while VideoMAE (VM) requires larger p.
5 CONCLUSION
Early action prediction is essential for timely decision making in safety-critical domains. This work identified the main components of successful early action prediction system. We introduced novel training framework that samples observation ratios in order to adapt the model to variable context length. Unlike the previous best method, this strategy enables training single model that requires 9x less compute and excels across all observation ratios. We further improved our base- line model by jointly training classification from forecasted and oracle features. Finally, we have proposed a training optimization that removes the visually repetitive half of the inputs, thus halv- ing the training memory. Our results demonstrate that training can be significantly simplified and still outperform the previous state-of-the-as art SSv2, NTU60 and UCF101 using more affordable hardware. Future research directions include unsupervised training and finding a unified method for both action anticipation and early action prediction.
10

---

6
ACKNOWLEDGMENTS
This work has been supported by Croatian Recovery and Resilience Fund NextGenerationEU (grant C1.4 R5-12.01.0001) and the Advanced computing service provided by the University of Zagreb University Computing Centre SRCE. We thank Ivan Grubišié for carefully reading the manuscript and providing constructive suggestions and insightful comments. We would like to ex- press our deep gratitude our beloved late professor, Siniša Segvić. As our mentor, he offered invaluable insight, direction and critical feedback. His intellectual leadership provided a foundation for this work, shaping its development and refinement. We remain sincerely grateful not only for his scientific vision, but also for his kindness, support and lasting influence on both this research and our professional growth.
REFERENCES
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Paltern Analysis and Machine Intelligence, 41(2): 423-443, 2019.
Aaron F. Bobick and James W. Davis. The recognition of human movement using temporal tem- plates. IEEE Transactions on pattern analysis and machine 22 2001.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Repre- sentations, 2023.
Cynthia Breazeal. Emotion and sociable humanoid robots. Int. J. Hum. Comput. Stud., 59(1-2): 119-155, 2003.
Guglielmo Camporese, Alessandro Bergamo, Xunyu Lin, Joseph Tighe, and Davide Modolo. Early action recognition with action prototypes, 2023. URL
06598
Yu Cao, Daniel Barrett, Andrei Barbu, Siddharth Narayanaswamy, Haonan Yu, Aaron Michaux, Yuewei Lin, Sven Dickinson, Jeffrey Mark Siskind, and Song Wang. Recognize human activities from partially observed videos. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 2658-26655 2013.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6308, 2017.
Joao Carreira, Eric Noland, Andras Banki- Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint urXiv: 080810088000000 0400, 2018
Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Niiuma,, Kris Kitani, and László Jeni. Don't look twice: Faster video transformers with run-length tokenization. Advances in Neural Infor- mation Processing Systems, 2024.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma. Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocen- tric vision: Collection, pipeline and challenges for epic- ppic-kicchens-100 International Journal of Computer Vision, 130(1):33-55 2022.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16359, 2022.
Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch n' papckk: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36:2252-2211 2023.

---

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A scage hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icce, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2020.
Mohsen Fayyaz, Soroush Abhasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and Jürgen Gall. Adaptive token sam- pling for efficient vision transformers. In European conference on computer vision, pp. 396-414. Springer, 2022.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6202-6211, 2019.
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946-35958, 2022.
Basura Fernando and Samitha Herath. Anticipating human actions by correlating past with the future with jaccard similarity measures. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13224-13233, 2021.
Lin Geng Foo, Tianjiao Li, Hossein Rahmani, Qiuhong Ke, and Jun Liu. Era: Expert retrieval and assembly for early action prediction. In European Conference on Computer Vision, pp. 670-688. Springer, 2022.
Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes. Predicting the future: A jointly learnt model for action anticipation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5562-5571, 2019.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3354-3361. IEBE, 2012.
Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the IEEE/CVF international conference computer vision, pp. 13505-13515, 2021.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phat, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Agrim Gupta, Jiajun Wu, Jia Deng, and Fei-Fei Li. Siamese masked autoencoders. Advances in Neural Information Processing Systems, 36:40676-40693, 2023.
Christopher G. Harris and Mike Stephens. A combined corner and edge detector. In Christopher J. Taylor (ed.), Proceedings of the Alvey Vision Conference, AVC 1988, Manchester, UK, September, 1988, pp. 1-6. Alvey Vision Club, 1988.
Joakim Bruslund Haurum, Sergio Escalera, Graham W Taylor, and Thomas B Moeslund. Which kens to use? investigating token reduction in vision transformers. In Proceedings the IEEF/CVF International Conference on Computer Vision, pp. 773-783, 2023.
the
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.
12

---

Jian-Fang Hu, Wei-Shi Zheng, Lianyang Ma. Gang Wang, Jianhuang Lai, and Jianguo Zhang. Early action prediction by soft regression. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 41(11):2568-22 2583, 2019. doi: 10. 109/TPAMM.2018.86622777
Sunil Hwang, Jachong Yoon, Youngwan Lee, and Sung Ju Hwang. Everest: Efficient masked video autoencoder by removing redundant spatiotemporal tokens. In International Conference on Ma- chine Learning, 2024.
Robert A. Jacobs, Michael Jordan, Steven J. Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 1991.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 35(1):221-231, 2012.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1732, 2014.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint vrXiv:1705.06950. 2017.
Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Bo- qing Gong. Movinets: Mobile video networks for efficient video recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16020-16030, 2021.
Yu Kong, Dmitry Kit, and Yun Fu. A discriminative model with multiple temporal scales for action prediction. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 596-611. Springer, 2014.
Yu Kong, Zhiqiang Tao, and Yun Fu. Deep sequential context networks for action prediction. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1473-1481, 2017.
Yu Kong, Shangqian Gao, Bin Sun, and Yun Fu. Action prediction from videos via memorizing hard- to-predict samples. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018a.
Yu Kong, Zhiqiang Tao, and Yun Fu. Adversarial action prediction networks. IEEE transactions pattern analysis and machine intelligence, 42(3):539-555 -53,,33,93 2018b.
Dong Hoon Lee and Seunghoon Hong. Learning to merge tokens via decoupled embedding for efficient vision transformers. In Conference on Neural Information Processing Systems, 2024.
Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uni- former: Unified transformer for efficient spatiotemporal representation learning. arXiv preprint WwXiv:2201.04670 2022a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Limin Wang, and Yu Qiao. Uni- formerv2: Spatiotemporal learning by arming image vits with video uniformer. arXiv preprint 1tXiv:2211.00552 2022b.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-effien efficient video foundation models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19948-19988 19960, 2023.
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong. Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp. 4804 4814, 2022c.
13

---

Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. In International Conference Learning Representations, 2022. /
Jy=BjyywnXXVV
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7083-7093, 2019.
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C Kot. Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding. IEEE transactions on pattern 26000)22162221 2019.
analysis and machine intelligence,
Xiaoli Liu, Jianqin Yin, Di Guo, and Huaping Liu. Rich action semantic consistent knowledge for early action prediction. IEEE Transactions Image Processing, 33:479-492, 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608v 2016.
Shugao Ma, Leonid Sigal, and Stan Sclaroff. Learning activity progression in Istms for activity detection and early detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1942-19500 2016.
Guoliang Pang. Xionghui Wang, Jianfang Hu, Qing Zhang, and Wei-Shi Zheng. Dbdnet: Learning bi-directional dynamics for early action prediction. In IJCAI, pp. 897-903, 2019.
AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova. Rethinking video vits: Sparse video tubes for joint image and video learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2214 2224,, 2023.
Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, Xiang Wang, Yuchuan Wang, Yiliang Lv, Changxin Gao, and Nong Sang. Mar: Masked autoencoders for efficient action recognition. IEEE Transac- fions Multimedia, 26:218 2023.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 14:13337-33949 2021.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 3505-3506, 2020.
Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Ag- garwal, Arkabandhu Chowdhury, Omid Poursaced, Judy Hoffman, et al. Hiera: A hierarchical vision transformer without the bells-and-whistee In International conference on machine learn- PMLR, 2023.
ing, pp.
Michael S Ryoo. Human activity prediction: Early recognition of ongoing activities from streaming videos. In 2011 international conference on computer vision, pp. 1043. IEEE, 2011.
Mohammad Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Basura Fernando, Lars Petersson, and Lars Andersson. Encouraging Istms to anticipate actions very early. Proceedings of the IEEE International Conference Computer Vision, pp. 280-289, 2017.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recogni- tion in videos. In Zoubin Ghahramani. Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q Weinberger (eds.). Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 668-576 2014.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset 101 human actions classes from videos in the wild. arXiv preprint arXiv: /2 222204022 2012.
14

---

Siddharth Srivastava and Gaurav Sharma. Omnivec2-a novel transformer based network for large scale multimodal and multitask learning. In Proceedings of the IEEE/CVE conference on com- puter vision and pattern recognition, pp. 27412-27424, 2024.
Alexandros Stergiou and Dima Damen. The wisdom of crowds: Temporal progressive attention for early action prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 14709-14719, June 2023.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances in neural information process- ing systems, 15:10078-10093, 2022.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489-4497, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pp. 98-106, 2016.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pp. 20-36. Springer, 2016.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 14549 14560,14500010000 June 2023.
Xionghui Wang, Jian-Fang Hu, Jian-Huang Lai, Jianguo Zhang, and Wei-Shi Zheng. Progressive teacher-student learning for early action prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Christopher Richard Wren, Ali Azarbayejani, Trevor Darrell, and Alex Paul Pentland. Pfinder: Real-time tracking of the human body. IEEE Transactions on pattern analysis and machine intelligence, 19(7):780-785, 1997.
Xinxiao Wu, Ruiqi Wang, Jingyi Hou, Hanxi Lin, and Jiebo Luo. Spatial-temporal relation reason- ing for action prediction videos. International Journal of Computer Vision, 129(5): 4845 2021a.
Xinxiao Wu, Jianwei Zhao, and Ruiqi Wang, Anticipating future relations via graph growing for action prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.
2021b.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification. Proceedings of the European conference computer vision (ECCV), pp. 305-321, 2018.
Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10809- 10018, 2022.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez- Pazz mixup: Beyond empiri- cal risk minimization. In 6th International Conference on Learning Representations. ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
He Zhao and Richard P Wildes. Spatiotemporal feature residual propagation for action prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7003-7012, 2019.
15

---

Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representa- tions (ICLR), 2022.
16

---

A APPENDIX
This appendix is organized as follows. We begin by discussing the method limitations. Next, we validate the decoder depth and measure the sensitivity to training with different random seeds. Fur- thermore, we demonstrate the effectiveness of the proposed framework using a different backbone. Additionally, we present comprehensive results of our method across all observation ratio values p. Most of these results are presented in the main paper. However, due to limited space, the main paper does not contain results under all observations ratios. Finally, we present additional analysis of the baseline EAST, on UCF101 dataset, along with additional qualitative examples.
A.1 LIMITATIONS
While EAST improves training efficiency and predictive performance, several limitations remain. Since the proposed token masking benefits training more than inference, the inference speed of ViT encoders limits real-time applications at today's hardware. Although we moved the needle towards practical
cases by training single model agnostic observation ratios, the model still requires a GPU to operate near real time. Moreover, our encoder does not perform causal inference, which necessitates sliding-window inference over the temporal dimension. This introduces two challenges: i) the minimum decision latency is bounded by the window length T, and ii) it prevents streaming inference, which would better capture natural temporal progression and long-term context. Note that evaluating streaming approaches is currently infeasible due to short video duration in existing early action prediction benchmarks.
A.2 ABLATION ON DECODER DEPTHI
Table 10 validates the number of transformer blocks in the decoder D. We evaluate depths of 1, and 12 blocks. The experiments show that decoder depth an important design choice. The decoder with 4 layers consistently achieves the best accuracy, improving by 0.1 pp over both and 12 layers on the SSv2 dataset. On the UCF101 dataset, the advantage is more pronounced, with improvements of 0.8 pp over layer and 1.3 pp over 12 layers. A 4 layer decoder expressive enough to transform the encoded features into accurate predictions, yet not too complex to avoid potential overfitting.
Table 10: Top-1 accuracy of EAST on SSv2 and UCF101 for three different decoder depths: 1, and 12. The best results are in bold.
Observation ratio p
Dataset
Depth
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
avg
25.4
29.9
34.1
41.2
48.6
55.1
59.5
63.0
64.0
46.8
SSv2
25.6
30.1
34.5
41.6
49.0
55.2
59.4
63.0
64.0
46.9
12
25.6
30.0
34.2
41.4
48.7
55.1
59.6
62.7
64.1
46.8
90.5
92.4
93.1
94.3
94.7
95.0
95.6
95.4
95.5
94.1
UCF101
91.3
93.2
93.8
94.7
95.5
96.1
96.4
96.5
96.5
94.9
12
90.3
92.0
92.8
93.6
94.3
94.6
94.9
94.9
94.9
93.6
A.3 ROBUSTNESS OF EAST TO RANDOM SEED
Our main results in Section use a fixed seed within single training run. We have found this to be common practice prior work. Nonetheless, demonstrate that our method is not sensitive to random seed selection. We perform additional training runs on Something Sooiiinn--omthinngg v2 with two more random seeds. Table [1 reports mean over three runs to confirm low sensitivity to randomness in training.
A.4 VALIDATION OF EAST USING A MOVINET BACKBONE
The first row in Table 12 shows the accuracy of an encoder-only model with a ViNi backbone. The second Table 12 shows that training encoder-only EAST with MoViNet backbone
17

---

Table 11: Top-1 accuracy (%) of EAST over all observation ratios for the SSv2, reported as the istd over three random seeds.
Observation ratio
Dataset 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
SSv2
25.5-0.2 29.8+0.3
34.2=0.3
40.1 48. 48.640.4 55.0
59.3
62.7
63.
63.7.0.3
Table 12: Contributions of the proposed losses and modules to UCF101 validation accuracy with MoViNet encoder. D denotes the choice of the decoder, id denotes an identity mapping (encoder- only), whereas usees the proposed decoder D.
Co greed
D p
0.1
P
0.2
P
0.3
P
0.4
p
0.5
p 0.6
p= 0.7
p= 0.8
p 0.9
avg
id
88.3
90.4
91.2
91.4
92.1
92.5
92.7
92.8
92.9
91.6
/
/
id
88.7
90.7
91.8
92.6
93.8
94.4
94.3
94.4
94.5
92.8
/
93.8
94.7
95.5
96.1
96.4
96.5
96.5
94.9
using both Cared and Coracte gains additional pp. Training encoder-decoder model using Cered and Conale (cf. MoVINa from Table 4 further improves average accuracy by 2.1 pp. The results highlight the benefits of training using the proposed compound loss in both cases, regardless of the backbone. Note that training with Mo limits the batch size to 32 since token masking is not applicable. In comparison, ViT-B/16 supports larger batch size of 128 the same GPUs.
A.5 EAST RESULTS PER P ON NTU60, SSSUB21 AND EK-100
Table 13 reports I accuracy EAST obtains at each observation ratio the NTU60, SSsub21 and EK-100 datasets. We provide a detailed performance comparison across the full range of evaluated observation ratios.
A.6 EAST ABLATION RESULTS ACROSS ALL OBSERVATION RATIOS
Table 14 shows the accuracy of EAST for every observation ratio p when using different decoders. Direct decoder Dair shows consistent improvement for all observation ratios in comparisons to au- toregressive decoder Dar. On average, the direct decoder yields a 0.3 pp improvement in accuracy.
Table 15 shows the accuracy EAST obtains when training with Le loss in conjunction with the proposed classification losses. We notice only marginal accuracy increase of 0.1 pp for 0.1. At other observation ratios, there is no benefit of using the L2 loss. On average, using only the classification losses improves accuracy by 0.3 pp.
Table 16 shows the accuracy of EAST for every observation ratio p when we use one classification separate classification heads. We notice minimal gain in accuracy of 0.1 pp for
head and when observation ratio p = 0.1. Using single classification head clearly improves the average accuracy by 0.3 pp.
A.7 PERFORMANCE OF EAST VS SINGLE MODEL FOR SINGLE p
Table 17 shows that training one model for each p can match occasionally surpass perfor- mance at its own observation ratio However, its accuracy deteriorates noticeably when applied to other ratios, with the decline becoming ing ratio. In contrast, EAST maintains consistently strong performance across all observation ratios, demonstrating greater robustness to changes in p.
severe the evaluation ratio diverges from the train-
18

---

Table 13: Extended Tables and from the main paper. Top-1 accuracy (%) of EAST over all observation ratios for the NTU60, SSsub21 and EK-100 datasets.
Observation ratio
Dataset
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
NTU60
31.2
49.6
69.4
81.3
86.2
87.6
87.9
88.0
87.9
SSsub21
40.8
44.7
51.2
59.2
66.4
72.0
75.8
78.3
79.3
EK-100 All Action
20.4
23.3
25.4
27.0
28.1
29.0
29.7
29.9
29.6
EK- 100 All Noun
31.1
34.0
35.5
37.3
38.2
39.3
39,9
40.5
40.1
EK- EK-100AAL
47.2
52.1
55.0
56.7
58.4
59.1
59.5
59.8
58.4
Table 14:
Extended Table 9
a) from the main paper. Top-1 accuracy of EAST on SSv2 for each p.
Direct decoder Dair consistenly outperforms autoregressive decoder Dar.
Observation ratio p
D
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
avg
Diir
25.6
30.1
34.5
41.6
49.0
55.2
59.4
63.0
64.0
46.9
Dar
25.0
29.5
34.2
40.9
48.1
54.5
58.8
62.4
63.3
46.3
b) from the main paper. Top-1 accuracy EAST on SSv2 over all reported observation ratios when using La in addition classification loss.
Table 15: Extended Table
Observation ratio p
L2
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
avg
x
25.6
30.1
34.5
41.6
49.0
55.2
59.4
63.0
64.0
46.9
25.7
29.8
34.4
40.9
48.2
54.7
59.1
62.6
63.7
46.6
Extended Table 9 c) from the main paper. Top-1 accuracy EAST on SSv2 over all reported observation ratios when using shared classification head vs separate classification heads for different set of features.
Table 16:
Observation ratio p
# cls h
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
25.6
30.1
34.5
41.6
49.0
55.2
59.4
63.0
64.0
2
25.7
30.0
34.3
41.0
48.3
54.7
59.2
62.5
63.5
Table 17: Numerical values for the Figure 2 in the main paper. Top-1 accuracy on SSv2 over all observation ratios when we train one model for each vs EAST. Results for the matching training are shown bold.
Observation ratio p
model
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
EAST
25.6
30.1
34.5
41.6
49.0
55.2
59.4
63.0
64.0
0.1
27.1
27.4
29.2
29.2
27.1
27.1
24.3
22.0
20.1
0.2
24.7
32.5
33.0
35.9
37.8
37.8
36.2
33.5
31.0
P 0.3
22.2
30.2
34.9
39.6
42.7
44.1
43.1
42.6
39.0
0.4
21.0
28.0
34.1
41.1
46.5
49.9
50.9
50.1
48.4
P 0.5
18.4
25.4
31.6
40.3
50.1
53.8
56.9
55.4
55.0
0.6
15.7
22.0
27.8
44.6
47.0
54.5
58.5
58.5
60.1
0.7
13.0
19.0
25.3
42.3
44.7
53.9
59.6
62.7
62.7
0.8
10.7
16.1
19.8
31.0
41.7
51.6
58.8
63.7
64.7
0.9
9.7
15.9
19.1
28.6
38.8
49.3
57.3
64.0
64.9
19

---

Table 18: Extended Table from the main paper. Top-1 accuracy of EAST on NTU60 over all reported observation ratios for different masking setups. M .MMAR and MM denote difference masking, Running Cell masking and random masking, respectively. denotes percentage of masked tokens. w/o denotes no masking of video frames. TFLOP denotes the number of floating point operations. Peak mem. denotes the maximum amount of GPU memory allocated any point during execution.
Observation ratio p
Masking
0.1
0.2
0.3
0.4
0.5
0.7
0.8
0.9
avg
TFLOP peak mem. (GB)
M rand
23.3
36.5
56.3
70.3
76.5
77.9
78.4
78.5
78.5
64.0
0.24
10.4
k 75
0.7G
27.3
40.9
59.4
72.2
78.0
79.3
79.7
79.8
79.8
66.3
28.4
46.4
65.9
78.3
84.0
85.6
86.1
86.1
86.1
71.9
00.75
rand
28.6
45.7
66.1
78.3
83.5
84.6
85.0
85.1
85.0 71.3
0.5
19.2
0.5
M MMAR
31.3
47.8
67.1
79.0
83.8
85.4
85.8
85.8
85.8 72.4
0.5
M
31.2
49.6
69.4
81.3
86.2
87.6
87.9
88.0
87.9
74.3
1.0.5
Mrand
2 25
31.1
48.3
67.9
79.8
85.1 86.8
87.2 87.3 87.3 73.4
0.8
27.9
M MAAR
31.5
48.1
67.8
79.8
84.9
86.6
86.9
86.9 86.9 73.3
0.25
31.9
51.0
70.8
81.9
86.7
88.3
88.6
88.7 88.6 75.2
0.25
w/o mask
32.6 50.7
70.3
82.1
86.9
88.1 88.4 88.5
88.5 75.1
1.1
36.7
A.8
RESULTS OF DIFFERENT MASKING SETUPS FOR EACH P
Table 18 shows that chosen masking strategy M - demonstrates a clear and consistent ad- vantage across observation ratios p. By selectively retaining the most informative tokens, effectively balances predictive accuracy and computational cost. This approach serves optimal middle ground, delivering strong performance while avoiding the excessive resource demands.
A.9 ADDITIONAL ANALYSIS OF THE OVERLOOKED BASELINE EAST. ON UCF101
Table 19 presents additional evaluation of EAST on UCF101. The results show that VideoMAE reaches competitive accu- racy for higher p. This trend is present in Table 6 on SSv2. Additional analy- sis Table 17 clarifies why VideoMAE achieves competitive results at higher ob- servation ratios. The table contains mod- els specifically trained at one observation ratio We can observe that these mod- els excel around y they trained at, whereas their accuracy drops when moving away from that specific p. Since VideoMAE is model specialized for 1.0, observe the same behavior: VideoMAE performs best around observation ratio it optimized for, i.e. at higher p values.
Table 19: Top-1 UCF101 accuracy over all observa- tion ratios. VideoMAE denotes pre-trained ViT-B/I6 model performance finetuned for action classification. EAST& trains ViT-B/16 with our proposed sampling without decoder.
Observation ratio p
Method
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
VideoMAE 67.6 77.3 80.5 82.2 84.5 84.5 84.9 85.0 85.0 EAST: 79.5 82.4 83.7 84.3 84.6 84.7 85.0 85.0 85.0
A. ADDITIONAL QUALITATIVE EXAMPLES
Additional qualitative examples are presented in Figure 4 highlighting the ability of EAST to make correct predictions starting from Lifting something with something on it (row 3) and Spinning something that stops (row 4), the VideoMAE is able to make the correct decision relatively early. How- ever, for examples like Opening something (row and Dropping something into something (row 2), VideoMAE needs to observe roughly 70 90% of the video to make re- liable classification. In some instances, like Closing something (row 5), VideoMAE still fails
0.1. We observe that for examples such as
20

---

to predict the correct class even after observing 90% of the video. On the other hand, EAST can make the correct decision even for the smallest observation ratio p = 0.1.
0.3
0.5
0.7
0.9
VM: Pretending to
VM: Pretending to open
YM: Pretending
upen
VM: Pretending
open
VM: Opening
somechiag
souchiing
somnthhin
sometiing
Souichhing
EAST: EAST: Opening
Opening
RAST: Opening
EAST: Opening
EAST: Opening
something
somelhing
something
something
something
0.1
0.3 Showing
0.5
== 0.7
0.9
Showling
VM: Dropping something
VM: Dropping something
empty X
empty X
EAST: Dropping
Dropping
Drupping
Dropping
Dropping
thing
thing
0.5
0.7
0,9
VME: Moving
Filting
VML something
VM: Litting
VM: Litting
thing X
somecing 7
sonnething
something
1
EAST: L Lifting- something with something it
EAST:Lifiie something
EAST: Lifting something
EAST: Liftng something with something on it
EAST Lilling something
with something It
with something on it
with something on
0.1
0.3
0.5
0.7
0.9
VM: Twnching
part of
YM: Pushinng something
Vpinninging something
VM: Spinning
something
YM: Spinning something
something
sppins
that stops
thai stops
that stops
EAST: Spinning
Some-
EAST: Spinning
RAST: Spinning
Some-
EAST: Spinning
somec-
RAST: Spinning
Somme-
thing that stops
thing that stops
thing that stops
thing that stops
thing that stops
0.1
0.3
0.5
0.7
0.9
VM: Holding
VM: Pretending to close
VM: Opening
VM: Moving part some-
VM: Moving part some
something
sounething
thing
Cloding
Closing
Closing
Figure 4: Qualitative analysis SSv2. We show the last frame within observation ratios. At the specified observation ratio X and / denote false and true model predictions, respectively.
21


Reviews:
[{"review_id": "cdO8v9taxP", "reviewer": "Reviewer_oxAh", "summary": "The paper introduces EAST (Early Action prediction Sampling strategy with Token masking), an encoder-decoder method for early action prediction. To improve classification performance, the encoder is trained considering both the previous and all the frames of the current action and employing a shared classifier. Moreover, training is performed by considering different observation ratios via sampling. This helps in having a single model able to handle various observation ratios. Finally, a token masking strategy is employed that leads to almost identical performance while only using 50% of the tokens, significantly improving the training efficiency.", "questions": "- What is the inference time of the proposed method?\n- Have inference time masking strategies been considered?\n- Why only sample at fixed increments of 0.1 and not uniformly in (0,1)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is interesting and well-justified. The joint training of the encoder on both previous and future frames is intuitive and appears to contribute positively to the performance.\n- The proposed training strategy, which considers random sampling of action observation ratio, makes the method robust against different observation ratios at test time. By using a single model for different observation ratios, the equivalent training time is significantly reduced.\n- The proposed token masking strategy further improves training efficiency by using up to 50% fewer tokens without any significant performance degradation.\n- Multiple ablations are considered, including the choice of the decoder, the use of shared vs separate classifiers, and various combinations of the loss functions.", "weaknesses": "- Although the introduction highlights the importance of inference time efficiency in various applications, only the training time and efficiency are discussed in the text. It is crucial to provide more details regarding the inference time of the proposed method.\n- Token masking is frequently considered in similar applications. Hence, although it most certainly helps in reducing the training efficiency, it cannot be considered a main contribution introduced in this work.\n- Related to both previous comments, masking at inference time is not considered in this work. It would be interesting to explore also this direction, which could contribute to the reduction of the processing time also during inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZlXOAhgV2N", "reviewer": "Reviewer_SbPM", "summary": "The paper addresses early-action prediction, where the model must recognise an action from only a partially observed video. The authors propose EAST, a training framework with three main ideas:\n\n1.  Randomised observation-ratio sampling: During training, the method samples an observation ratio ρ ∈ {0.1,…,0.9} and splits each video into an “observed” prefix Vᵒ and an “unobserved” suffix Vᵘ. A single model is then trained to operate across all ρ, rather than training one specialised model per ρ (as prior work like TemPr typically does).\n\n2.  Compound loss on present vs. full video: The model is trained both to classify from the partial clip (via a decoder that forecasts future features) and to classify the full clip (oracle). Optimising both prediction logits (from predicted future features) and oracle logits (from full context) is argued to make the encoder’s features discriminative even under very small observation ratios.\n\n3. Token masking for efficiency: The authors introduce a difference-based token masking strategy that drops ~50% of spatio-temporal tokens deemed redundant across time. This reduces training memory and wall-clock time while preserving accuracy.\nEAST reports strong gains on NTU60, Something-Something v2 (SSv2 and SSsub21), and UCF101, on both VideoMAE and MoViNet backbones. The paper also includes ablations: (i) with/without the decoder, (ii) with/without each loss term, (iii) different masking ratios, and (iv) per-ρ specialised models vs. one unified model. The authors claim that EAST achieves new state-of-the-art results while requiring fewer GPUs and no per-ρ retraining.", "questions": "1. Clarify the temporal sampling: Please explicitly define T, T_d, Vᵒ, Vᵘ, and the operator ∥. When you say “The sampled clip V = Vᵒ ∥ Vᵘ consists of 2T evenly spaced frames centered at ρ·T_d,” does that mean |Vᵒ| = |Vᵘ| = T, with Vᵒ covering frames strictly before ρ·T_d and Vᵘ strictly after ρ·T_d, and no overlap? Or does “centered” mean you actually take a symmetric window around ρ·T_d and then label the pre-cut portion “observed” and the post-cut portion “future”? A precise definition is important for reproducibility.\n\n\n2. Generalisation across ρ: You state that “training at fixed observation ratios produces models that are suboptimal at other values of ρ.” You partially support this claim with Figure 2 (specialist models vs EAST). Could you quantify how much performance drops when a ρ-specialist is evaluated out-of-distribution (e.g., a model trained at ρ=0.3 evaluated at ρ=0.7)? Are all these models trained with exactly the same backbone, number of epochs, and data seen per step, so that the comparison is fair?\n\n3. Loss design and theory: The compound loss combines L_pred (classification on forecasted features from partial observation) and L_oracle (classification on oracle full-clip features). Can you provide intuition for why this helps so much at low observation ratios (ρ=0.1, 0.2)? For example, is the model learning to align early-frame features with full-clip features, or is it essentially acting like a teacher-student relationship inside the same network? Did you try scheduling L_oracle to zero out later in training (to avoid overfitting to full context)?\n\n4. Token masking: The masking method selects tubelets with a large temporal difference based on an L1 distance across time. Can you include a short figure or pseudocode showing this ranking process? The current text around Eqs. (5–7) is quite dense and abruptly introduces p, d, and N_t. At inference time, do you also apply masking? If not, is there any distribution shift between masked training and unmasked inference? Does masking ever remove fast motion in the observed portion, hurting the ability to predict very early?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Overall, even if the ideas are incremental individually, the combination is interesting and could serve as a practical baseline for early-action prediction. Below are the main strengths of this paper.\n\n1. Practical relevance and framing. Early action prediction is important for any system that must react before an action is complete (e.g., driving, human-robot interaction). The paper motivates this scenario well.\n\n2. Single model across all observation ratios. Prior work often trains a different model for each observation ratio ρ. This is expensive and awkward for deployment. EAST’s randomised sampling of ρ at training time directly addresses that, and Figure 2 suggests that such a unified model generalises well across all ρ, whereas per-ρ specialists do not transfer. This is an important engineering insight for the field.\n\n3. Compound loss improves robustness at low observation ratios. The combination of predicting from partially observed clips and supervising with an “oracle” view of the full clip encourages discriminative features even at 10%–20% observation. The ablations (Table 7) confirm that using both loss terms (L_pred and L_oracle) improves accuracy over using either alone.\n\n4. Training-time efficiency via token masking. The masking procedure drops ~50% of “low-difference” tubelets and roughly halves peak memory usage, while maintaining accuracy (Table 6). This is a concrete, useful contribution, and the authors do report memory/TFLOP numbers and epoch times.\n\n5. Strong experimental results. EAST achieves new state-of-the-art top-1 accuracy across a range of observation ratios on SSv2, SSsub21, NTU60, and UCF101, using both VideoMAE and MoViNet backbones. Comparisons include modern baselines like TemPr (CVPR 2023), ERA (ECCV 2022), and RACK (TIP 2023).\n\n6. Reproducibility and ablations. The paper discloses hyperparameters, training schedule, backbone initialisations (VideoMAE pretraining, MoViNet), and even wall-clock timings. The ablation test: decoder types (direct vs autoregressive), loss design, shared vs separate classifier, masking ratios, etc.", "weaknesses": "1. Clarity and notation issues in the method section: the construction of Vᵒ (observed) and Vᵘ (unobserved) around a sampled ratio ρ is central, but the notation is underspecified. The text states: “The sampled clip V = Vᵒ ∥ Vᵘ consists of 2T evenly spaced frames centered at ρ·T_d,” implying |Vᵒ| = |Vᵘ| = T. However, T is not defined before use, and “centered” conflicts slightly with “concatenation of Vᵒ and Vᵘ with no gap.” This makes it unnecessarily hard to reproduce the exact temporal sampling logic.\nThe encoder paragraph (“Tokenizer T splits the input clip frames into N_t … tubelets of size d×p×p=2×16×16 … The transformer encoder V extracts features…”) mixes general notation and hard-coded hyperparameters in a colloquial way. Key variables (T, p, d, N_t) are introduced abruptly, and not all are defined rigorously. For a core technical section of an ICLR paper, this could be tighter.\n\n2. Limited theoretical insight: The paper argues that combining (i) a loss on predicted future features from partial clips (L_pred) and (ii) a loss on oracle full-clip features (L_oracle) makes the encoder discriminative under all observation ratios. While plausible, this is only described qualitatively; no analysis (e.g., feature visualisations, calibration curves, entropy at low ρ) is provided.\n\n3. Missing qualitative results: The paper does not include any qualitative timeline-style figure showing how the predicted class probabilities evolve as more of the video is seen, nor any visual comparison to ground truth. This is fairly standard in early \naction anticipation literature to illustrate how “early” the model can correctly commit to an action. Adding even one such figure would greatly improve the interpretability and trustworthiness of the qualitative claims.\n\n4. Comparisons to the most recent literature: The comparisons include TemPr (CVPR 2023), ERA (ECCV 2022), and RACK (TIP 2023), which are strong baselines. However, there are also recent approaches that model early action recognition via prototypical representations or transformer decoders and report top-1 accuracy vs. observation ratio on SSv2 / SSsub21 / EPIC-Kitchens in late 2023 (e.g., “Early Action Recognition with Action Prototypes”, a.k.a. Early-ViT-style models). These methods also aim at efficient online recognition from partial clips. EAST does not cite or compare against that family, and it would strengthen the impact claim to position EAST relative to those more recent models.\n\n5. Evaluation metrics: The paper reports only top-1 accuracy across multiple observation ratios ρ, which is indeed standard in early action prediction work such as TemPr and other pre-2024 baselines. Still, reporting additional metrics like top-5 accuracy, AUC across ρ, or calibration / time-to-correct-prediction curves would make the evaluation more comprehensive, especially since the method is motivated by safety-critical, real-time applications.\n\n6. Slightly overstated claims: The authors state they achieve “9× less compute,” “2× faster training,” and training on “consumer-grade hardware.” These claims are mostly motivated by: \n\n   a.  Not needing to train nine separate models (one per ρ). \n\n   b.  ~2× memory savings from their masking scheme (Table 6). 3. Wall-clock per-epoch measurements (80s/epoch vs 173s/epoch for TemPr on the same 4× RTX A6000 setup).\n\nWhile these are promising, the paper could be more precise: RTX A6000s are still high-end GPUs; and the ‘9× less compute’ claim is inferred rather than measured in total GPU-hours. This is more of a tone issue than a scientific flaw, but worth noting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uTc0yZPJjQ", "reviewer": "Reviewer_9qPd", "summary": "This paper presents EAST, a simple yet effective framework for early action prediction (EAP) that enables a single model to generalize across multiple observation ratios. The method introduces a randomized temporal sampling strategy to jointly train on both observed and unobserved video segments, combined with a token masking technique that removes redundant patches for more efficient training. Extensive experiments on NTU60, Something-Something V2, and UCF101 demonstrate state-of-the-art accuracy while reducing training time and memory by half.", "questions": "Please refer to the weakness section above.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "S1. Simple yet effective: The proposed sampling and masking strategies are conceptually simple but yield strong performance gains across benchmarks.\n\nS2. In-depth analysis: Extensive ablation studies support each design choice and show consistent improvements.\n\nS3. Comprehensive evaluation: EAST is evaluated on multiple datasets, demonstrating generality and efficiency.", "weaknesses": "W1. Weak motivation & significance: The paper does not clearly elaborate which specific limitations in prior EAP methods EAST addresses or why its components are crucial. While the sampling strategy is intuitive and well-justified, the roles of the compound forecasting loss and token masking are underexplained. Moreover, the significance of the tackled problem is not sufficiently emphasized. Clarifying both the motivation and broader impact of the proposed framework, along with the necessity of each component, would substantially strengthen the paper’s contribution.\n\nW2. Outdated baselines: Many comparisons are against older methods; TemPr is the only recent baseline, but its architectural and training differences make fair comparison difficult. Additional controlled analysis on temporally challenging datasets (e.g., Something-Something) would clarify the contribution.\n\nW3. Missing inference speed comparison: Since EAP is latency-sensitive, comparing inference speed with prior works (e.g., TemPr, ERA) would strengthen the claims.\n\nW4. Limited masking comparison: Token masking is only compared with random masking. Including modern masking methods such as VideoMAE [a] or MAR [b] would better validate the approach.\n\nW5. Missing results on Epic-Kitchens 100: Although Epic-Kitchens 100 is a standard benchmark for action anticipation and applicable to EAP (as demonstrated in TemPr), the paper omits it. Including this dataset would better validate generalization in real-world scenarios.\n\n[a] Tong et al., “VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training,” NeurIPS, 2022. \\\n[b] Qing et al., “MAR: Masked Autoencoders for Efficient Action Recognition,” Transcations on Multimedia, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tONFX0x8Ci", "reviewer": "Reviewer_mSTD", "summary": "This paper reseaches on action early recogntion (or action prediction, coined by ealier work) in videos. Insteading of pursuing complicated techniques, this paper actually advocates and demonstrates that simple model architecture and training schema would be sufficient to have good results, as well as be sufficient to accomendate a varitety of inference requirements (e.g., observation ratios in this use case). Strong empirical results support the claim.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A. Very impressive evaluation results - consistently improving on popular benchmarks by notable margin.\n\nB. The overall approach bears a good balance of simplicity - the model architecture being transformers (i.e., a very general and prior-free model family), the optimization strategy (i.e., the training loss is a standard cross-entropy loss), the design for being agnostic against observation ratios with one unified approach, and lastly the hyper-parameters also being straightforward and easy to implement.", "weaknesses": "A. This review has strong concern on the assumptions incoporated and implemented in the token masking module, which is proposed for the purpose of the effienct model design - repetitive frame masking. This strategy, though indeed reduces FLOPS and speeds up training/inference, pre-assumes that similar tokens should be ignored and largely different tokens in feature space should be attended to more, which is arguable when it comes to a boarder scope of action early recongtion task. Imagine fast action video clips such as \"world chase tag\" competition where the frames are mostly motion intense, have much less idle motions and even the idle period is quite important. It is very unlikely that the 50% frame masking strategy would hold all essential information. Even more, let's assume 50% masking rate is appropriate, the masking decision comes from the token-wise similarity captured by feature space difference. This strategy has inherent flaws, such as big background motion will be reckoned as more valuable compared to small actor related motion under current fashsion. From the experimental results, it seems that using full frame (or token) still bring the best results, which makes the masking less critical. To sum up, this reviewer appreciate the efforts on making slim model, but disagree on doing so depending on any prior assumptions, such as feature space difference.   \n\nB. Can the authors further clarify the protocol difference between the proposed EAST method and the most recent work, e.g., Tempr? This reviewer found an evaluation tip, i.e., multi-crop inference, a training tip, i.e., data augmentaion, and a scaled learning rate schedule, in the paper and wonders if the comparison method follow the same procedure. Sometimes, training and evaluation details can lead to big difference. It would be nice to keep the protocol the same on both side. This reviewer conjure that difference in optimization could has strong impact on final results.\n\nC. How is the VideoMAE trained in Table 5? Is it just using the observed frames at a certain ratio to predict the action label? Is it also a observation ratio agnostic model? Why the results indicate the impact of \"sampling strategy\"? What is the loss and input here?\n\nD. This reviewer is particularly interested in Table 5 and would like to observe more on the performance gain when being compared to VideoMAE from other dataset. Actually, VideoMAE is more comparable to EAST than other early action recognition work (e.g., Tempr) in terms of model architecture similarity and loss objectives. It will be a big help if authors can add such comparison to all benchmarks and spot any interesting patterns. One thing noticed by this reviewer from the current Table 5 is that the 0.9 observation ratio accuracy from EAST_{epson} is deteriorated, i.e., 60.7 vs. 63.4. And this makes reviewer wonder if current VideoMAE is biased on later frames (i.e., focus on 80th - 90th percent frames of a video clip). If so, it seems logically reasonable that the proposed training strategy depicted in figure 1 resolves such bias to some extent, or say learn a more generalized classifier across time - The L^{pred} will object when L^{oracle} optimize later frames too much. The learning dynamic here can be a very interesting investigation. Maybe plotting the L^{oracle} of 80th-90th percent frames vs L^{pred} on the validation set during training would be insightful, which also fits the taste of ICLR (more machine learning analysis).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)