PrAViC: Probabilistic Adaptation Framework for Real-Time Video Classification

## Abstract

Abstract Video processing is generally divided into two main categories: processing of the entire video, which typically yields optimal classification outcomes, and real-time processing, where the objective is to make a decision as promptly as possible. The latter is often driven by the need to identify rapidly potential critical or dangerous situations. These could include machine failure, traffic accidents, heart problems, or dangerous behavior. Although the models dedicated to the processing of entire videos are typically well-defined and clearly presented in the literature, this is not the case for online processing, where a plethora of hand-devised methods exist. To address this, we present PrAViC, a novel, unified, and theoretically-based adaptation framework for dealing with the online classification problem for video data. The initial phase of our study is to establish a robust mathematical foundation for the theory of classification of sequential data, with the potential to make a decision at an early stage. This allows us to construct a natural function that encourages the model to return an outcome much faster. The subsequent phase is to demonstrate a straightforward and readily implementable method for adapting offline models to online and recurrent operations. Finally, by comparing the proposed approach to the non-online state-of-the-art baseline, it is demonstrated that the use of PrAViC encourages the network to make earlier classification decisions without compromising accuracy.

## 1 Introduction

In recent years, there has been a notable increase in the utilization of convolutional neural networks (CNNs) across a range of fields where the capacity to make expeditious decisions could be crucial, including medicine [, ], human action recognition (HAR)  [, ], and autonomous driving []. However, despite the growing prevalence of CNNs in these domains, there remains a lack of a unified approach to the problem of making early decisions based solely on the initial frames.

Conversely, numerous offline approaches have been proposed (see, e.g.,  [, , , , ]) to address the problem of video data classification. However, these models usually require access to entire videos, which precludes their real-time applicability. Although certain techniques have been developed to facilitate the adaptation of offline models to the online domain (e.g., [, ]), there remains a need for the development of more generalizable solutions that can accommodate diverse forms of data.

To address the aforementioned gap, we propose a novel probabilistic adaptation framework for real-time video classification (PrAViC). In contrast to traditional methodologies, our approach allows for the adoption of existing three-dimensional convolutional models, wherein subtle adjustments are made to the architectures in order to retain the utility of the pre-trained classification model weights. Moreover, our strategy encompasses the recursive application of this adapted model in the real-time analysis

Figure: Figure 1: Progress of the model’s early decision function during training.
Refer to caption: extracted/5672533/images/squeezenet_NE_lambda_0.5.png

of video data streams. This pioneering approach not only slashes the model training time by leveraging pre-existing weights but also unlocks the potential for recursive utilization. The ramifications of this advancement extend across various sectors including industry, medicine, and public safety, where swift real-time analysis is paramount for informed decision-making and proactive measures.
In the experimental study, we demonstrate the efficacy of our approach when applied to a selection of state-of-the-art offline models trained on three real-world datasets, including two publicly available datasets, UCF101 [] and EgoGesture [], as well as a closed-access real Ultrasound dataset, comprising Doppler ultra-317 sound images representing short-axis and suprasternal views of newborn hearts.
Furthermore, we introduce an innovative function that allows the model to make earlier exits (decisions) when sufficient evidence is gathered. The effect of such a function is presented in Fig. 1. By incorporating this function into the objective, the model optimizes the timing of its decisions, enhancing efficiency with a slight loss of accuracy.

In conclusion, our contribution can be summarized as follows:

- •
we introduce PrAViC, a novel, unified, and theoretically-based probabilistic adaptation framework for online classification of video data, which has the potential to make a decision at an early stage, thus improving the efficiency of the classification process,
- •
we demonstrate that our solution provides a straightforward and readily implementable method for adapting offline video classification models to online and recurrent use,
- •
we conduct experiments which indicate that the proposed approach is capable of facilitating earlier classification decisions without compromising accuracy when compared to selected non-online state-of-the-art solutions.

## 2 Related works

While video-based 3D networks have been widely studied in offline settings  [, , , , ], developing online models remains still a challenge. Such online models are able to real-time classification, enabling applications such as emergency situations detection or medical diagnostics.

#### Adaptations 2D models for online video data

Given that video consists of consecutive frames, each frame can be classified individually using 2D CNN models  []. Such an approach represents an online technique and works well as real-time classification, due to a low number of parameters. This method however might not be optimal due to the absence of temporal information from the entire video, potentially resulting in misclassifications, such as predicting "sitting down" when the action involves transitioning from sitting to standing.
To develop a model capable of processing online data in real-time, researchers often opt for 2D networks supplemented with additional mechanisms to manage temporal data  [, , , , ]. The authors of [] adopted a temporal shift module (TSM). By shifting part of the channels along the temporal dimension, TSM can help capture temporal relationships.

#### Online 3D models for real-time usage

Since it was proven [], that 3D CNNs
achieve better accuracies compared to 2D CNNs for video
classification task, novel approaches employ differing versions of 3D convolutional kernels. Several works proposed dedicated architectures of online 3D networks  [, , , ], which are supposed to operate in real-time. Combining 3D CNN architectures with various long short-term memory models is another commonly utilized approach.  [] introduce a DNN merging 3D DenseNet variants and BiLSTM.
In  [] R2plus1D and ConvLSTM are combined in a parallel module. The proposed network utilizes the attention mechanism to extract the features that need attention in the channel and the spatial axes. Another approach  [], combines a vision transformer for human pose estimation with a CNN-BiLSTM network for spatio-temporal modeling within keypoints sequences. Attention mechanisms additionally with transformer layers were also applied in  [].
Converting various well-known resource-efficient 2D CNNs into 3D CNNs [] is another proposed approach. The most closely related work to ours is  [], which explores the adaptation of 3D CNNs (especially 3D ResNet family models) for online video stream processing. Köpüklü, Okan, et al. [] employ no temporal downsampling combined with storing intermediate volumes of the architecture in a cache for use during inference.

## 3 Probabilistic Adaptation Framework for Real-time Video Classification (PrAViC)

This section presents the details of the proposed PrAViC model (Probabilistic Adaptation Framework for Real-time Video Classification). For the reader’s convenience, the problem of video classification is divided into offline and online. In the offline case, where the entire video is available, classification can be performed by processing the entire video. In the online case, where consecutive images are obtained, the objective is to make a decision using only a partial subset of the potentially available frames.

#### Offline case

We commence with a description of the model for the offline case. For the sake of simplicity, we limit our discussion to the case of binary classification. We are given a video $V=[V_{0},\ldots,V_{n}]$, which is represented as a sequence of images $V_{i}$ (frames). Then, the network $\phi:\mathcal{V}\to[0,1]$ returns the (soft) probability that $V$ belongs to the positive class. The final decision is based on the threshold $\tau$, where typically $\tau=1/2$. Thus if $\phi(V)\geq\tau$, we conclude that the class of $V$ is positive.

Let us consider the case where the soft probability is unknown, but
for every threshold value $\tau$ we can deduce whether $\phi(V)\leq\tau$. In this scenario, one can obtain the soft probability $\phi(V)$ as follows:

$$ $\begin{array}[]{c}\phi(V)=\text{Prob}(\phi(V)\leq\tau:\tau\sim\mathrm{unif}_{[ 0,1]}).\end{array}$ (1) $$

This can be interpreted as selecting a random threshold value $\tau$ from the interval $[0,1]$ and calculating the probability of being below the threshold.

The subsequent paragraph addresses the question of how the offline model can be applied to the online procedure.

#### Online (early exit) procedure

We describe the standard general setting for the online early exit model. We assume that the frames arrive consecutively (on occasion, we permit them to arrive in groups of, e.g., two, four, or eight). Thereafter, given a trained classification network $\phi$ (as described in the previous paragraph), we fix a threshold $\tau\in[0,1]$ and proceed with the following steps:

- 1.
start with $k=0$;
- 2.
load $V_{k}$ and compute $p_{k}=\phi([V_{0},\ldots,V_{k}])$;
- 3.
if $p_{k}\geq\tau$, then return class 1 (positive) for $V=[V_{0},\ldots,V_{n}]$; else put $k=k+1$;
- 4.
if $k=n+1$, then stop the algorithm and return class 0 (negative); else return to step 2.

The objective of the aforementioned procedure is to allow making the decision of $V$ being in a positive class before loading all frames from $V$. It should be noted that if we operate in the offline mode, where we have access to all data, the above algorithm can be constrained to computing $p=\max\{p_{1},\ldots,p_{k}\}$, and then determining that the class of $V$ is negative if $p<\tau$, or positive otherwise.

The underlying concept of our proposed solution is that of a continuous probabilistic model for the decision threshold of activity detection. This is discussed in detail in the following paragraph.

#### Probabilistic model

First, observe that once the aforementioned procedure is completed, we are only aware of the decision that has been made, but we lack the information necessary to calculate the probability of the given outcome. Without this information, it is not possible to use BCE loss and fine-tune the model. To address this issue, we apply the probabilistic concept from Eq. (1),
which allows us to calculate the soft probability of the decision, given the knowledge that the decision was made for an arbitrary threshold. Then we get that the probability that $V$ has the positive class in the online case is given by

$$ $\begin{array}[]{c}p(V)=\max(p_{0},\ldots,p_{n}),\text{ where }p_{i}=\phi([V_{0 },\ldots,V_{i}]).\end{array}$ (2) $$

This is of paramount importance insofar as our objective is for the model to make more timely decisions (see Fig. 2).

Figure: Figure 2: The left image illustrates a probabilistic model where the outcome is determined by selecting the maximum value of $p_{i}$ (0.89 in this case). The right image shows an alternative approach where the model can terminate earlier based on available probabilities. For the example provided, the model exits with a probability of 0.42 at the 1st step, 0.18 at the 3rd step, and 0.29 at the 4th step.
Refer to caption: x1.png

#### Expected time of early exit

To proceed to the probabilistic approach, we first calculate the expected time our model makes an early exit. For a fixed video $V$, $T_{V}$ denotes the random variable that returns the time we have made an early exit and $\infty$ when no early exit was made. We are interested in calculating the function $\mathrm{Exit}(V)$, which represents the expected time of early exit (provided that it has been made).
In order to achieve this, we define the random variable $W=T_{V}|T_{V}<\infty$ with values from the set $\{0,1,\ldots,n\}$ as follows: we draw a random variable $\tau$ uniformly from the interval $[0,\max_{i}p_{i}]$, and for the given value of $\tau$, we return the first index $k\in[0,n]$ for which $p_{k}\geq\tau$.
Then we have

$$ $\begin{array}[]{c}\mathbb{E}(W)=p(W\geq 1)+\ldots+p(W\geq n)=n-P(V\leq 0)- \ldots-P(V\leq n-1).\end{array}$ (3) $$

Finally, we get

$$ $\begin{array}[]{c}\mathrm{Exit}(V)=n-\frac{1}{\max p_{i}}\sum\limits_{t=0}^{n- 1}\max\limits_{i=0..t}p_{i}\in[0,n]\;\;\;\text{and}\;\;\;\mathrm{NE}(V)=\frac{ 1}{n}\mathrm{Exit}(V)\in[0,1].\end{array}$ (4) $$

The function $\mathrm{NE}$ is defined as a normalization of $\mathrm{Exit}(V)$ with respect to the number of frames. It returns 0 0 if we have made the exit (with probability one) on the frame $V_{0}$, and $1$ if we cannot exit before $V_{n}$ and the probability of exit in $V_{n}$ is positive.

#### Loss function for PrAViC

Now we will describe how to incorporate the $\mathrm{NE}$ function into the loss function in a way that encourages the model to exit earlier. However, we do not want to stimulate the model too much so as not to provoke wrong decisions. To do this, we set a parameter $\lambda\in[0,1]$ that tells us how large a percentage of confidence we are willing to potentially sacrifice in order to make the decision early. Consider the videos $\bar{V},\tilde{V}$ from the positive class, where $p(\bar{V})=1,\mathrm{NE}(\bar{V})=1$ and $p(\tilde{V})=\tau,\mathrm{NE}(\tilde{V})=0$. Our goal is to add an additional part to the loss function in such a way that the loss of $V$ and $\tilde{V}$ is equal. Obviously
$\log(\lambda+(1-\lambda)\mathrm{NE}(\bar{V}))=\log(\lambda+(1-\lambda)\mathrm{
NE}(\tilde{V}))$.

Consequently, we define the objective for image $V$ with class $y\in\{0,1\}$ as follows:

$$ $\mathrm{loss}(V,y)=\mathrm{BCE_loss}(V,y)+y\log(\lambda+(1-\lambda)\mathrm{NE }(V)).$ (5) $$

It is evident that for videos belonging to the negative class, the loss function remains the standard BCE loss. Conversely, for videos $\bar{V}$ and $\tilde{V}$ discussed above, we have $\mathrm{loss}(\bar{V},1)=\mathrm{loss}(\tilde{V},1)$. It is also noteworthy that for $\lambda=1$, the additional part of the loss function is equal to 0, which discourages the model from making early decisions. As the value of $\lambda$ approaches 0, the model is encouraged to make decisions as rapidly as possible, even if this results in a loss of accuracy.

## 4 Architecture of the model

In this section, we introduce the architecture of our proposed approach, detailing the modifications and enhancements made to the CNN-3D network. Our approach involves specific alterations to key layers, including the convolution layers and batch normalization, as well as a unique method for processing the network’s head. Our architectural modifications aim to leverage the strengths of traditional CNN-3D networks while addressing specific challenges related to depth processing and feature extraction. The following sections provide a detailed breakdown of each component and its role within our approach.

#### Architecture enabling fine-tuning

We outline the modifications made to the classic 3D CNN architecture to adapt it to our approach. While most components of a standard CNN architecture remain unchanged, we specifically alter the 3D convolution processing, batch normalization, and layer pooling. For 3D convolutions, we modify only those layers where the kernel size responsible for the depth (i.e., processing movie frames) is greater than 1. Our modification ensures that the kernels do not extend to the last deep channel. To achieve this, we replicate the input boundary on the front side before performing the multiplication operation, as illustrated in Fig. 3. For pooling layers, our modification involves replicating only the first depth channel.

Figure: Figure 3: The picture illustrates the mechanism of the classic 3D convolution processing (left) with parameters: padding 1, stride 2, and kernel size 3, compared to our modified convolution processing (right). Note that the proposed change in input processing does not alter the kernel weights.
Refer to caption: x2.png

The next essential mechanism in CNN networks is batch normalization []. It involves calculating the mean and standard deviation for individual dimensions within mini-batches and training gamma and beta parameters during network training. This process ensures consistent and stable training across the network by maintaining the integrity of feature scaling and normalization, which is crucial for effective spatiotemporal pattern learning. In our approach, we modify this mechanism by statically determining the number of depth channels from which the statistics (mean and standard deviation) will be calculated. These statistics are then used to transform the remaining depth channels.

It’s important to note that through these operations, each layer of our model retains information from preceding depth channels only. Additionally, our approach possesses a unique property: when provided with inputs of the same movie, where one contains $k$ frames and the other $n$ frames ($k<n$, with the second input being an extension of the first by $n-k$ frames), the network produces identical outputs, constricted to the dimensions of the output from the $k$ first depth channels of inputs.

Figure: Figure 4: The picture depicts a diagram of the proposed approach during the training phase (left) and the recursive evaluation phase (right). In the model evaluation phase, the red area retains the computations from previous frames in individual network layers, while the yellow area represents the calculations for the newly added frame. Without retaining the computations in the red area, they would need to be recomputed, resulting in a longer evaluation process. This procedure allows for the rapid processing of new frames in an online manner.
Refer to caption: x3.png

#### Head design

Here, we describe the modifications to the head of the CNN network to enable online training and recursive evaluation as we show in Fig. 4. Typically, in the case of a CNN, the head consists of the final linear layer, so in our approach, we leave the head as a linear layer, but we will process the output from the last convolutional layer differently than it is done in a classic CNN network. We assume that $v_{0},\ldots,v_{n}\in\mathbb{R}^{D}$ represent the outputs from the last convolutional and pooling layers, considering only the height and width dimensions while retaining the depth dimension as it is after the final convolutional layer. With this representation, following the standard offline approach, we perform one of the mean aggregation of representations relative to time $t\in[0,\ldots,n]$:

$$ $\begin{array}[]{c}w_{t}=\tfrac{1}{t+1}\sum\limits_{i=0}^{t}v_{i}.\end{array}$ (6) $$

Using the above aggregations $w_{t}$, we process each separately through a linear layer $h$ followed by the sigmoid $\sigma$ function to obtain $p_{t}=\sigma(h(w_{t}))$. The final decision of the model is determined by the formula(^1^11One can also use soft maximum given by $p=(\tfrac{1}{n+1}\sum_{i=0}^{n}p_{i}^{s})^{1/s}$ for parameter $s\gg 1$, which performs usually better in gradient optimization applications. Note that for the limiting $s=\infty$, we obtain a hard maximum.) $p=\max\limits_{t=0..n}p_{t}$. The source code is available at [https://github.com/...](https://github.com/...)

## 5 Experiments

In this section, we detail the experimental setup and present the results that validate the effectiveness of our proposed approach. We utilized three datasets for testing: two publicly available (UCF101 [] and EgoGesture []) and one closed-access (Ultrasound). Various architectures were employed in these experiments, each of which is described in the respective subsections. The numerical experiments were conducted using NVIDIA RTX 4090 and NVIDIA A100 40GB GPUs.

### 5.1 Comparison on UCF101 dataset

In this subsection we compare PrAViC with the baseline non-online approaches, ResNet-3D-18 (R3D-18)  [] and Separable-3D-CNN (S3D) [].

**Table 1: Classification results on UCF101 dataset. R3D-18 and S3D are baseline non-online models. Our modifications applied to these models are indicated by PrAViC(name of the baseline model). PrAViC^⋆(name of the baseline model) indicates additional custom loss function (5). The third column presents the average decisive frame number i.e. number of the frame, on which the model made the classifying decision. The addition of a custom loss function forced both models to make decisions much earlier that models with categorical cross-entropy.**
| Model | offline | online |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | Best Accuracy | Best Time |  |  |  |
| R3D-18 | S3D | PrAViC(R3D-18) | PrAViC(S3D) | PrAViC^⋆(R3D-18) | PrAViC^⋆(S3D) |  |
| Accuracy(%) | 94.41 | 96.45 | 92.63 | 92.08 | 79.82 | 92.93 |
| Avg decisive<br>frame | 16 | 16 | 11.71 | 9.71 | 7.06 | 6.03 |

We conduct experiments on widely used video benchmark, UCF101  []. This dataset is composed of 13320 videos from 101 action categories. These 101 categories fall into 5 subgroups: Body motion, Human-human interactions, Human-object interactions, Playing musical instruments, and Sports. Together, the video clips span over 27 hours. They are collected from YouTube, and uniformly recorded at 25 frames per second with a resolution of $320\!\times\!240$. Each video clip is labeled with a single action class. The dataset is partitioned into three sets: training, validation, and testing in a ratio of 6:1:1. From each video clip we extract 16 consecutive frames, starting with a random frame of a whole video clip, which enables such operation. For the R3D-18 model, we resize each frame to resolution $128\!\times\!171$, and randomly crop each frame to size $112\!\times\!112$. For the S3D model, we resize each frame to a resolution $256\!\times\!256$ and randomly crop each frame to size $224\!\times\!224$.

First, we consider classic offline models for video classification: R3D-18 and S3D, which have been pre-trained on Kinetics-400  [] dataset. We replaced number of outputs in the last layers to meet our classification problem with 101 distinct classes. Training took part in two stages. In the beginning, we trained just the last, classification layer for 5 epochs. After that, we fine-tune the entire network for 10 epochs. The batch size was set to 16 for both models. We use SGD optimizer. Learning rates for the first stage and second stage for R3D-18 were set to 0.01 and 0.0002, respectively. For S3D model: 0.01 and 0.0002, respectively. Cross entropy was the loss function.

Next, we modified R3D-18 and S3D networks as described in Sec. 4. Due to these changes, offline models became online. Same as previously, we replaced a number of outputs in the last layer of each model and training took part in two stages. In the first stage, we trained the last layer for 60 epochs. Fine-tuning was set up to 80 epochs. The batch size was set to 16 for both models. We use SGD optimizer. Learning rates for the first stage and second stage for modified R3D-18 were set to 0.002. For the S3D model: 0.015 and 0.0001, respectively. We report accuracy and number of decisive frames, i.e. number of frames, which on the first model was able to make a decision. Lastly, we applied a custom loss function as defined in  (5). Our goal was to force the model to make an earlier decision. Various $\lambda$ parameters were evaluated, starting with 0.1 up to 1.

Figure: Figure 5: Comparison of accuracies and average decisive frame numbers depending on custom loss function parameter $\lambda$. First row presents results for PrAViC^⋆(R3D-18) and second row for PrAViC^⋆(S3D). The model has a tendency to delay decisions and have higher accuracy as the $\lambda$ parameter values increase, contrasting with quicker decision-making and lower accuracy observed for lower parameter values. Observe that encouraging the model to make early decisions results in a small loss in accuracy.
Refer to caption: extracted/5672533/images/ucf_acc_lambda.png

In Tab. 1 we present classification results of baseline non-online models (R3D and S3D) and their online equivalents (PrAViC(R3D-18), PrAViC(S3D), PrAViC^⋆(R3D-18), PrAViC^⋆(S3D)) on UCF101 dataset. Our modified versions of baseline models are referred to as PrAViC(name of the baseline model), while PrAViC^⋆(name of the baseline model) denote additional custom loss function  (5). Accuracies for PrAViC^⋆(name of the baseline model) models are chosen for the best $\lambda$ parameter (for accuracies for different $\lambda$ parameters see Fig. 5). Moreover, the best, across different $\lambda$ parameters, the average decisive frame is listed for online models. By adding a custom loss function for PrAViC(name of the baseline model) networks, we achieved a decision speed approximately 1.5 times faster. For PrAViC^⋆(S3D)) model accuracy does not decrease significantly.

Fig. 5 shows accuracies and averages decisive frame numbers for $\lambda$ parameter varying from 0.1 up to 1. We observe increasing both accuracy and average decisive frame number, as the $\lambda$ parameter rises.

### 5.2 Comparison on EgoGesture dataset

In this experiment, we harnessed a collection of trained models: 3D-SqueezeNet, 3D-ShuffleNetV1, 3D-ShuffleNetV2, and 3D-MobileNetV2, as outlined in [].
These models were trained on the EgoGesture dataset [], tailored for egocentric hand gesture recognition.

In setting up this experiment, we employ the learned model weights [] on the data, initially training new model heads, followed by a light retraining of the entire model, with or without incorporating an additional cost function to promote earlier detection. The architectures used in our approach are the same as those of the base models, with modifications detailed in Sec. 4 of the paper. We utilize stochastic gradient descent (SGD) with standard categorical cross-entropy loss for training the architectures. The momentum, damping, and weight decay are configured as 0.9, 0.9, and 0.001, respectively. The network learning rate is initialized at 0.1, 0.05, and 0.01, then decreased by a factor of 3 with a factor of 0.1 when the validation loss reaches convergence.

During model training, input clips are selected from random time positions within the video clip. If the video comprises fewer frames than the input size, loop padding is implemented. For input into the network, we utilize clips with multiples of 16 frames. Specifically, for 3D-MobileNetV2, we use 128 frames, while for the remaining models, we utilize 192 frames. This approach is adopted to yield 8 probability values $p_{i}$ at the network output for the MobileNet network and 12 for the other networks. A single input clip has dimensions of $3\!\times\!n\!\times\!112\!\times\!112$, where $n$ represents the number of frames, varying based on the models used (as described above).

**Table 2: The table presents the Top 1 (A@1) and Top 5 (A@5) accuracy results for both validation (val) and test sets of EgoGesture data. Base models are the best models trained on this dataset available at [https://github.com/ahmetgunduz/Real-time-GesRec](https://github.com/ahmetgunduz/Real-time-GesRec). The proposed PrAViC approaches are denoted as ’our’ and ’our^⋆’, where the star indicates the use of an additional cost function for earlier decision-making. Our approach, utilizing the same architectures with modifications described in Sec. 4, shows improved performance in most cases and supports online recursive usage (see Fig. 4).**
| Metric | Data<br>Split | SqueezeNet | ShuffleNet | ShuffleNetV2 | MobileNetV2 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | base | our | our^⋆ | base | our | our^⋆ | base | our | our^⋆ | base | our | our^⋆ |  |
| A@1 | test | 88.23 | 91.84 | 90.86 | 89.91 | 86.05 | 84.35 | 90.46 | 91.44 | 90.73 | 90.31 | 91.38 | 91.84 |
| val | 84.25 | 87.14 | 86.16 | 85.77 | 82.12 | 80.01 | 85.83 | 86.60 | 85.17 | 86.28 | 86.56 | 87.50 |  |
| A@5 | test | 97.63 | 99.20 | 99.16 | 98.28 | 99.10 | 99.18 | 98.36 | 99.29 | 99.20 | 98.22 | 99.20 | 99.33 |
| val | 95.88 | 94.58 | 95.66 | 96.40 | 96.40 | 96.91 | 96.81 | 96.14 | 96.72 | 96.91 | 97.37 | 97.53 |  |

Tab. 2 provides a detailed performance comparison between the proposed PrAViC approach and the base models retrained on the EgoGesture dataset, highlighting Top 1 and Top 5 accuracy results for both validation and test sets. As observed from the table, the PrAViC approach generally outperforms the base models across most metrics. This improvement is evident in both Top 1 and Top 5 accuracy for both validation and test sets. Furthermore, our approach offers the added advantage of being applicable in an online, recursive manner, providing enhanced functionality and efficiency in practical applications (see Fig. 4 for a visual representation). These results underscore the effectiveness of the proposed modifications and highlight the potential of our approach to deliver superior performance in online gesture recognition tasks.

### 5.3 Comparison on Ultrasound dataset

In the subsequent experiment, the developed method was tested on a real dataset of Doppler ultrasound images representing short-axis and suprasternal views of newborns’ hearts. These recordings were obtained as part of an ongoing scientific research project involving pediatric cardiologists, with the consent of the newborns’ parents. During the acquisition process, a total of 18,365 ultrasound recordings were collected.
The standard video analysis offline model trained on this dataset was modified by changing the classification layer (head) as well as the convolutional layers, in particular batch norm. These changes allowed for subsequent frames to be added sequentially while the model was running. These changes required fine-tuning of the model. Since the classification layer was completely new, fine-tuning started with 5 epochs of learning only this layer with a learning rate of $10^{-5}$. Subsequent training of this model for 10 epochs, with the same learning rate, but this time with training of all changed layers, allowed to achieve $94\%$. Standard Cross-entropy as a cost function was used in both stages.

Using our own cost function provided in Eq. (5), which prefers to classify as a class 1 element as quickly as possible, if it belongs to it, allowed us to achieve slightly lower accuracy as in the standard model, so approximately $90\%$ regardless of the lambda parameter as in Tab. 3. Due to the model’s use of remembered history, the model’s accuracy was tested by introducing 16 frames into the model, i.e. the number needed for each convolutional layer inside the model to have at least one output representation other than padding. Only subsequent frames were added one at a time.

**Table 3: Classification on Ultrasound Dataset with depending on custom loss function $\lambda$ parameter. The model does not show much difference in accuracy depending on the $\lambda$ parameter, because most decisions are made after processing the initial 16 frames anyway.**
| $\bm{\lambda}$ | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Accuracy(%) | 86.95 | 88.53 | 88.53 | 90.90 | 90.90 | 91.30 | 91.30 | 90.90 | 90.90 | 88.93 |

The model was then repeatedly evaluated on the selected video with different frame delivery parameters. The average time from 100 evaluations was taken as the result of the experiment for each set of parameters. The first parameter was the total number of frames delivered after all steps were completed.
Due to the growing history in memory, it was expected that the increase in time would not be linear, i.e. 2 times as many frames should result in an evaluation more than 2 times longer. The experimental results confirmed this thesis. On the other hand, not single frames but entire batches can be loaded into the model. For this reason, in the second experiment, the first experiment was repeated, but instead of a single frame, 4 or 8 frames were fed to the model in each step. Thus, the history stored in the model worked in the same way as before, but the number of reads and writes to it, was reduced. Due to the expected operation of the model on devices with lower performance, e.g. mobile devices, the first part of the tests were carried out on a CPU.
The experiments were also repeated using the CUDA architecture. The results from both devices are presented in Fig. 6.

Figure: Figure 6: Model evaluation on time. Each line corresponds with the device used for evaluation and the number of frames inserted into the model in one step.
Refer to caption: x4.png

The conducted experiments have demonstrated the efficacy of the developed methodology in identifying congenital heart defects (CHDs) in neonates through ultrasound imaging. CHDs, encompassing conditions such as Tetralogy of Fallot, Hypoplastic Left Heart Syndrome, and Ventricular Septal Defect, pose formidable diagnostic hurdles owing to their intricate nature and the subtleties inherent in early cardiac anomalies. Regrettably, undetected instances of such defects represent a prominent contributor to neonatal mortality rates. The adapted video analysis model utilized in this investigation heralds a significant leap forward in the diagnostic realm pertaining to congenital heart defects.

## 6 Conclusions

In this work, we propose PrAViC, a
a general framework to automatically modify networks adapted to the processing of the whole video, to their online counterpart. To do this we introduce the probabilistic theoretical model behind online data processing. This allows us to define the expected exit time and use it as a component of the loss function to encourage the model to make early decisions.

#### Limitations

In PrAViC we use standard convolutions which have a time component. Although this allows the use of pre-trained networks, this does not allow the full use of the full frames of the video in the residual network, which can potentially lead to a small loss of accuracy.

#### Broader impact

Since the use of online analysis of video is gaining importance in medical and social applications, our method can be applied as an easy-to-use tool for researchers working in applied video processing.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness
soundness: 3
strengths: 1. The propose solution is reasonable and the paper is well written 
2. The experiment results are impressive, which may help for other sub-sequent tasks
summary: The paper proposed a method of an online video classification by using the early-exit mechanism. The solution is reasonable and the results is impressive.
weaknesses: 1. more baselines should be added to validate the effectiveness of the proposed approach. For example, some other on-line mechanisms, even the simple threshold one. 
2. Some discussion should be added to explain the intuition of the method, and more ablation study should be added to validate the findings, and the contributions.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - Eq.3 is defined and never used. Furthermore, it’s not clear how the expected value is computed as reported. Generally speaking, you define W, and never use it anywhere.
 - Eq.3 clearly reports $P(V\le c)$, which makes no sense, since $V$ is defined as a set of frames
 - It is not clear, and as previously stated, not defined, how Eq.3 and Eq.4 are defined. But more generally speaking, the whole “Expected time of early exit” section should be rewritten as definitely not clear, which is a problem since it lays the ground for the rest of the paper. Indeed, by the looks of things, what you are doing with Exit(V) is just the integral of the blue  histogram in Fig 3, though this is a way more intuitive way to explain the term, over the notation used in the section
 - in Eq.5 it is not clear why you decided to use a log scale for the regularization. More generally, the equation would benefit from a more extensive explanation, showing the intuition behind it
 - in line 375, it is said that accuracy increases, though in the S3D case, that’s opinable observing the plot. Ideally, if it’s still the case that the authors want to state that, please run a statistical analysis to check if any relevant trend can be seen in such a plot and report the p values somewhere in the paper.
 - in line 378, you state that you _compare_ your approach to many others, though, from Table 1, it seems more like that you add your approach on top of such methods. At no point, however, you state what you mean when you refer to “baseline” in Table 1
 - in table 1, ignoring that the authors never define what “baselines” means, there are some results that increase the accuracy by 10%, which is almost-99% error rate. however the results are compared to the setting lambda=1 which means that the loss is composed of only the BCE loss. However, that term is the main contribution of the paper. Thus, it’s not clear how it is possible to have such a bump in performance.
Writing concerns:
 - line 152 “This is useful as in the case of online...” might need a rephrasing
 - line 162, since you are using “=” for comparison and assignment, i’d suggest to use “:=” for assignment, just for clarity
 - line 234, you are using alpha instead of lambda
 - line 304 contains an empty link
 - Figure 5 contains a random “3" behind the word “sigmoid” on the right
 - line 383, there is a citep that should be a normal cite
 - the paper contains only very long main sections (\section). Instead, it’s highly suggested the usage of \subsections, \subsubsection, and so on to structure the paper in a more reasonable way
soundness: 2
strengths: - The related work section is well presented and easy to read
 - Section 3, up to “Expected time of early exit”, gives a clear overview of the problem that the paper wants to address
summary: This paper introduces a novel scheme for online video classification. Such a method is based on the training/finetuning of offline models with a regularization term that should drive the classification network to give early classification, defining a tradeoff between computation and accuracy. The method is tested by equipping previously presented architecture with such new loss.
weaknesses: - the paper is generally not clear from section 3 on
 - the proposed approach is never well defined. a custom loss function is proposed, but it is never said how to then use the model to do inference with early exit. Instead the paper would benefit from a comparison between a baseline model, with a fixed threshold, and the proposed finetuned ones, to compare the time the two models take to classify the video as 0/1 above the significance threshold picked
 - the theoretical part is highly confusing, with missing explanations and equations and quantities that are never used. in particular, all of these concerns are reported in the “Problems” section of this review.
 - the paper misses a section comparing the proposed approach to any baseline for online classification, even basic ones reported in “Online (early exit) procedure”
 - The paper is very practical, with little and very confusing theoretical contributions. For this reason, it is fundamental that the paper is presented with a reproducible code, which would also help the understanding of the not-so-clear “Expected time of early exit”. Particularly, ICRL strongly encourage to add a “Reproducibility Statement” (https://iclr.cc/Conferences/2025/AuthorGuide), and the paper is missing it.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: - According to the paper's premise of using the model for real-world situations, two analyses would have been interesting: analyzing the model's false positive rate and how it changes according to the lambda hyperparameter and the impact of the early exit on the overall prediction time (in seconds). 
- A term alpha appears in the middle of the text and it is never shown. I suppose that is a mistake and should be the hyperparameter lambda. (Above Eq. 5, Pg.5)
- In the abstract, the authors mention that they will evaluate three datasets with two of them publicly available. They mention four open datasets, as they also correctly explain in the experiments.
soundness: 3
strengths: - Good and helpful images.
- The probabilistic adaptation framework is a clever idea to handle the need for classification as fast as possible.
- The architectural modifications are simple and easy to implement in any offline CNN structure.
- The experiments cover a sufficient range of datasets and relevant 3D-CNN models to show the effectiveness of the proposed framework.
- Comparison with online and offline models.
- Another method for online video classification is used for comparison.
summary: This paper proposes a probabilistic adaptation framework for real-time video classification. The framework is a probabilistic approach that encourages the model to decide whether the instance belongs to the positive class as soon as possible. Besides, the authors also propose a series of adaptations to transform an offline video classification model into online classification. The authors evaluate the framework on four open datasets: UCF101, EgoGesture, Jester, and Kinetics-400, considering different 3D-CNN architectures. The experiments show that the proposed framework (and architectural modifications) can adapt offline models to online models and that the models can make decisions faster than offline models.
weaknesses: - The paper does not mention the SOTA models for video classification and action recognition, such as VideoMAE and other transformer-based architectures. Transformer models are a well-established SOTA, especially for some of the evaluated datasets.
- The paper discusses only 3D-CNNs and does not clarify such a limitation in the abstract or contributions. The framework's adaptations do not include different architectures, such as transformers or even CNN-LSTM.

Papers:
VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking
Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models
OmniVec2 - A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning

- The explanation of the Expected Time of Early Exit is not very clear. The term P(V = x) is used before explaining P in the text, making it unclear for the reader if they are referring to Video V or some property of it. (Eq. 3, Pg. 4)

- Authors do not provide a GitHub repo; only provide an indication that it will be released. It could be anonymized and provided.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: I incorporated my questions to the weaknesses section. Please see the above.
soundness: 2
strengths: - **Early classification:** PrAViC makes faster decisions by exiting early when confident, a crucial feature for applications like emergency detection, medical diagnostics, and autonomous systems.
- **Minimal accuracy loss:** The framework maintains accuracy comparable to offline models, even with early exits.
- **Adaptability:** PrAViC can be integrated with existing 3D CNN architectures and easily adapted for real-time operation with minimal modifications.
- **Extensive experiments:** The paper presents experiments across five real-world datasets.
- **Clear visual aids:** Figures such as Fig. 3, 4, and 5 effectively illustrate the proposed model, aiding comprehension.
summary: The authors propose PrAViC, a framework for early (online) video classification. PrAViC incorporates a stopping mechanism that allows the model to exit early and make decisions based on partial video frames, thereby improving classification speed while maintaining high accuracy. The framework tracks the maximum class probability during frame processing and exits once a certain confidence threshold is reached, predicting the class with the highest probability at the time of exit.

Please note that the following represents my initial impressions of the paper, and I am open to discussion. I welcome any corrections to potential misunderstandings.
weaknesses: The paper was an enjoyable read, but I found the following potential weaknesses and questions. I am willing to raise my score once these issues are addressed.  


**Lack of highly related baselines.**  
The work is positioned as a variant of Early Classification of Time Series (ECTS), especially given the focus on “real-time processing” and early decision-making. However, there is no mention or evaluation of ECTS algorithms [1, 2, 3], making it difficult to understand how PrAViC compares to existing work. ECTS literature often employs loss functions that balance classification accuracy and timeliness, much like the approach taken in Eq. (5).  

Several existing early classification models, such as video early classification [4], could be applied to the problem setting defined by the authors. Although many ECTS algorithms are feature-vector-based [5, 6, 7] and not intended for video stream per se, they can also be applied to the aggregated feature in Eq.(6) under PrAViC framework. 

To justify the novelty and effectiveness of their loss function, the authors should compare PrAViC against these established methods.  While the proposed framework demonstrates solid results across diverse real-world scenarios, evaluating against related baselines would strengthen its position.

[1] Xing+, Early prediction on time series: A nearest neighbor approach. IJCAI 2009  
[2] Gupta+, Approaches and applications of early classification of time series: A review. IEEE TAI 2020.  
[3] Mori+, Reliable early classification of time series based on discriminating the classes over time. DMKD 2016   
[4] Ma+, Learning activity progression in lstms for activity detection and early detection. CVPR 2016  
[5] Chen+, Early time series classification using TCN-transformer. ICCASIT 2022    
[6] Hartvigsen+, Adaptive-halting policy network for early classification. KDD 2019  
[7] Bilski+, CALIMERA: A new early time series classification method. Information Processing & Management 2023  


**Lack of proper evaluation for online classification.**   

While early exit or ECTS models should be evaluated on *both* classification performance and earliness (such as NET), some experiments report only accuracy (e.g., Table 1), or only PrAViC's NET (e.g., Table 2). Ideally, early exit models should not be evaluated based on a single point in the accuracy-NET space; rather, they should be assessed using a set of points that represent the tradeoff between accuracy and earliness (i.e., the Pareto front). Benchmarking the proposed model and the baselines across this tradeoff would provide a more comprehensive evaluation, showcasing how well the model balances early exits and accuracy, which is crucial for early classification tasks.  

Additionally, given the real-world datasets used, which often suffer from class imbalance, accuracy may be misleading [8]. More appropriate metrics would be balanced accuracy or mean macro-averaged recall, which would offer a clearer picture of the model’s performance across all classes.  

[8] Branco+, A survey of predictive modeling on imbalanced domains. ACM Comput. Surv 2016


**Potentially overstated theoretical contribution.**

The authors claim that PrAViC is a theoretically-based framework, contrasting it with "hand-devised methods." However, I found no formal theoretical guarantees such as convergence rates, statistical consistency, or classification-calibrated loss. If the "theoretically-based" claim stems solely from the thresholding mechanism using class probabilities, this risks misleading readers and undermining the significance of PrAViC’s novelty.

Furthermore, there’s no engagement with well-established literature on Sequential Hypothesis Testing [9, 10, 11], which already provides theoretical guarantees for similar problems. Including a comparison or discussion of this body of work would strengthen the theoretical claims of the paper.  

Thus, to substantiate the claim of being "theoretically-based," the authors could strengthen their work by providing formal guarantees, similar to what is done in Sequential Probability Ratio Test (SPRT) [9]. For example, SPRT guarantees the minimal number of samples (or equivalently, exit time) under a predefined error rate, assuming certain statistical conditions. If PrAViC could offer a theoretical guarantee of exit time (or the balance between exit time and error rate) under similar assumptions, it would provide a strong theoretical foundation and significantly enhance the contribution of the paper. Such guarantees would better differentiate PrAViC from "hand-devised" methods and provide formal support for its early exit mechanism.

[9] Wald, Sequential tests of statistical hypotheses. Annals of Mathematical Statistics, 1945  
[10] Tartakovsky. Asymptotic optimality of certain multihypothesis sequential tests: Non-i.i.d. case. Statistical Inference for Stochastic Process, 1998  
[11] Miyagawa+, The power of log-sum-exp: Sequential density ratio matrix estimation for speed-accuracy optimization. ICML, 2021  


**Inappropriate ablation study**  

The ablation study is not conducted in the conventional sense. Typically, I would expect the ablation study to evaluate the contribution of individual components (e.g., removing batch normalization, disabling the loss term). However, the authors introduce a new model, CoX3D, and conduct new experiments, which is not in line with what is usually meant by an ablation study. This adds confusion and detracts from the clarity of the experimental analysis.    


**Reproducibility concerns.**  

The code for PrAViC is mentioned but not provided (there’s a broken link to GitHub), raising concerns about reproducibility. Furthermore, the authors report only scalar accuracy values, without error bars or statistical tests, making it difficult to evaluate the robustness and generalizability of their results. For a complex training process, such as "light retraining" of models, more details should be provided to ensure that others can replicate the experiments.   


**Unclear PrAViC framework for multiclass classification.**

Section 3 formulates the problem in a binary classification setting, but it’s not clear how the framework extends to K-class classification (e.g., UCF101 with 101 classes). Are the authors treating it as a K+1-class problem (one background class and K event classes), or are they performing standard K-class classification without a background class? Clarifying this would enhance the understanding of the framework's flexibility in real-world, multiclass scenarios.  


**Unclear motivation and novelty.**   
The authors state that PrAViC is motivated by the need for "more generalizable solutions that can accommodate diverse forms of data." However, the terms "generalizable" and "diverse forms of data" are not clearly defined in the paper. It would be beneficial to clarify whether "generalizable" refers to performance across different datasets or some specific capability of PrAViC that makes it more adaptable than existing online classification methods.

Additionally, "diverse forms of data" seems to refer to the range of real-world datasets (video streams) used in the experiments. However, since the paper only addresses video classification, it would help to rephrase or elaborate on this claim to avoid confusion. Classical CNN-RNN architectures [12] already perform online video classification effectively, so it would also be useful to highlight the specific contributions PrAViC offers that go beyond these established methods.

[12] Karpathy+, Large-scale Video Classification with Convolutional Neural Networks, CVPR 2014  


**Minor comments**

- l.063 “two publicly available datasets” → should be "four"?  
- When I saw Fig.2 for the first time, I had no idea what does NE(V) stands for. I would add a definition or at least reference in the legend, but this may be a personal preference.  
- The term "unified" is used repeatedly, but it’s unclear what aspect of PrAViC is unified compared to the literature.  
- l.178 “BCE” loss is introduced without spelling out. I can tell it’s a binary cross entropy-loss, but please avoid letting a reader guess.  
- l.203~ “In this example, the model terminates with a probability of 0.42 at the 1st step, a probability of 0.18 at the 3rd step, and a probability of 0.29 at the 4th step.” → should be 0.60 at the 3rd, and 0.89 at the 4th?   
- Figure 6: The caption should specify which dataset is used (UCF101).
- Figure 6: Adding a horizontal line to show the baseline model performance would help clarify the comparison.

Meta-review: This paper was reviewed by four experts in the field and received 6, 3, 5, 3 as the ratings. The reviewers agreed that the proposed PrAViC method enables faster video classification by exiting early when confident, which can be useful in a variety of applications; it can be easily integrated with existing 3D CNN architectures, and has been validated on 5 real-world datasets. 

The reviewers raised concerns about the comparison baselines, stating that PrAViC has not been compared against related algorithms for Early Classification of Time Series (ECTS); it is difficult to assess the merit of the framework without an exhaustive comparison against the state-of-the-art methods. While the authors claimed that PrAViC is a theoretical framework, no formal theoretical guarantees (such as convergence rates, statistical consistency, or classification-calibrated loss) were provided in the paper. Concerns were also raised about the ablation studies, where new experiments were conducted with a new model, rather than the conventional method of evaluating the contribution of each individual component. The writing quality of the paper also needs to improve. The authors are also encouraged to open-source their implementation for better reproducibility (the provided GitHub link is broken).

The authors did not respond to the individual reviewer's comments. In light of the above discussions, we conclude that the paper is not ready for an ICLR  publication in its current form. While the paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.

Decision: Reject