STREAMER: Streaming Representation Learning and Event Segmentation in a Hierarchical Manner
Authors: Ramy Mounir
Sujal Vijayaraghavan
Sudeep Sarkar
Venue: NeurIPS 2023 poster (2023)
Keywords: predictive learning
hierarchical event segmentation
self-supervised learning
streaming processing
perceptual inputs
biologically-plausible.
TL;DR: A self-supervised method that processes perceptual input in a streaming manner and outputs a hierarchical structure of semantic event representations.

Abstract: We present a novel self-supervised approach for hierarchical representation learning and segmentation of perceptual inputs in a streaming fashion. Our research addresses how to semantically group streaming inputs into chunks at various levels of a hierarchy while simultaneously learning, for each chunk, robust global representations throughout the domain. To achieve this, we propose STREAMER, an architecture that is trained layer-by-layer, adapting to the complexity of the input domain. In our approach, each layer is trained with two primary objectives: making accurate predictions into the future and providing necessary information to other levels for achieving the same objective. The event hierarchy is constructed by detecting prediction error peaks at different levels, where a detected boundary triggers a bottom-up information flow. At an event boundary, the encoded representation of inputs at one layer becomes the input to a higher-level layer. Additionally, we design a communication module that facilitates top-down and bottom-up exchange of information during the prediction process. Notably, our model is fully self-supervised and trained in a streaming manner, enabling a single pass on the training data. This means that the model encounters each input only once and does not store the data. We evaluate the performance of our model on the egocentric EPIC-KITCHENS dataset, specifically focusing on temporal event segmentation. Furthermore, we conduct event retrieval experiments using the learned representations to demonstrate the high quality of our video event representations. Illustration videos and code are available on our project page: https://ramymounir.com/publications/streamer

STREAMER: Streaming Representation Learning and Event Segmentation in a Hierarchical Manner
Ramy Mounir Sujal Vijayaraghavan Sudeep Sarkar
Department of Computer Science and Engineering, University of South Florida, Tampa (ramy, sujal, sarkar] edu
Abstract
We present a novel self-supervised approach for hierarchical representation Iearn- ing and segmentation of perceptual inputs in streaming fashion. Our research addresses how to semantically group streaming inputs into chunks at various levels of hierarchy while simultaneously learning, for each chunk, robust global rep- resentations throughout the domain. To achieve this, we propose STREAMER, architecture that is trained layer-by-layer, adapting to the complexity of the input domain. In our approach, each layer is trained with two primary objectives: making accurate predictions into the future and providing necessary information other levels for achieving the same objective. The event hierarchy is constructed by detecting prediction error peaks at different levels, where detected bound- ary triggers a bottom-up information flow. At an event boundary, the encoded representation of inputs at one layer becomes the input higher-level layer. Additionally, we design a communication module that facilitates top-down and bottom-ut up exchange of information during the prediction process. Notably, our model is fully self-supervised and trained in streaming manner, enabling single pass on the training data. This means that the model encounters each input only once and does not store the data. We evaluate the performance of our model on the egocentric EPIÇ-KITCHENS dataset, specifically focusing on temporal event segmentation. Furthermore, we conduct event retrieval experiments using the learned representations to demonstrate the high quality of our video event representations. Illustration videos and code are available on our project page: attps://ramymount com coom//ublccationnn momns//tteame....
Computational theory
temporal event analysis, an event is defined as "a segment time that is perceived by an observer to have a beginning and an end" [59]. Events could be described by sequence of constituent events of relatively finer detail, thus forming a hierarchical structure. The end of an event and the beginning of the next is segmentation boundary, marking event transition. Segmentation boundaries in the lower levels of the hierarchy represent event transitions at relatively granular scales, whereas boundaries in higher levels denote higher-level event transitions.
We propose a structurally self-evolving model learn the hierarchical representation of such events in self-supervised streaming fashion through predictive learning. Structural evolution refers to the model's 's capability to create learnable layers ad hoc during training. One may argue that existing deep learning architectures are compositional in nature, where high-level features are composed of lower-level features, forming a hierarchy of features. However, it is important to distinguish between feature hierarchy and an event hierarchy: event hierarchy is similar to a part/whole hierarchy in the sense that each event has clear boundaries that reflect the beginning and the end coherent chunk of information. One may also view the hierarchy redundancy pooling mechanism, where
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Pour oll
Add chicken
Move chicken
Add chicken
Wash board
Move chicken
Ground truth
High level
STREAMER's output
Low level
Figure 1: Comparison of STREAMER's hierarchical output to single-level ground truth annotations from EPIC-KITCHENS, The ground truth contains redundant narrations for successive annotations (e.g., add chicken D; STREAMER identifies such instances as a single high level event (Narrations from ground truth)
information grouped as one event considered redundant for higher level and can be summarized into a single representation for higher-level processing.
Our model is capable of generating hierarchy of event segments (Figure 1) by learning unique semantic representations for each event type directly from video frames. This is achieved through predictive learning, which models the causal structure of events. These learned representations are expressive enough to enable video snippet retrieval across videos. Each level in the hierarchy selectively groups inputs from the level below to form coherent event representations, which are then sent to the level above. As a result, the hierarchy exhibits temporally aligned boundaries, with each level containing subset of the boundaries detected in the lower level.
As often prescribed [28, 24], we impose the following biologically-plausbi constraints on our learning algorithm:
1. The learning algorithm should be continuous and online. Most existing learning algo- rithms offer batch-basee offline learning. However, the learning in the neocortex occurs continuously streaming fashion while seeing each datapoint only once
2. The learning should involve the ability to make high-order predictions by "incorporating contextual information from the past. The network needs dynamically determine how much temporal context is needed to make the best predictions" [24] (Section 1.1)
3. Learning algorithms should be self-supervised and should not assume labels for training [37]; instead, they should be able to figure out the learning objective from patterns and causal structures within the data
4. The learning should stem from a universal general-purpon algorithm. This is supported by observations of the brain circuitry showing that all neocortical regions are doing the same task in repeated structure of cells [24]. Therefore, there should be no need for global loss function (i.e., end-to-end training with high-level labels); local learning rules should suffice (Section 2)
1.1 Predictive learning
Predictive learning refers to the brain's ability to generate predictions about future events based on past experiences. fundamental process in human cognition that guides perception, action, and thought [58, 31]. The discrepancy between the brain's predictions and the observed perceptual

---

inputs forms a useful training signal for optimizing cortical functions: if model can predict into the future, it implies that it has learned the underlying causal structure of the surrounding environment. Theories of cognition hypothesize that the brain only extracts and selects features from the previous context that help minimizing future prediction errors, thus making the sensory cortex optimized for prediction of future input [51]. A measure of intelligence can be formulated as the ability of a model to generate accurate, long-range future prediction [53].To this end, we design an architecture with the main goal of minimizing the prediction error, also referred to maximizing the model evidence in Bayesian inference according to the free energy principle [19, 18].
Event segmentation theory (EST) suggests that desirable properties such as event segmentation emerge as byproduct of minimizing the prediction loss [59]. Humans are capable of chunking streaming perceptual inputs into events (and chunking spatial regions into objects [14]) to allow for memory consolidation and event retrieval for better future predictions. EST breaks down streaming sensory input into chunks by detecting event boundaries as transient peaks in the prediction error. The detected boundaries trigger process of transitioning (i.e., shifting) to a new event model whereby the current event model is saved in the event schemata, and a different event model is retrieved, or a new one initialized to better explain the new observations. One challenge implementing a computational model of EST is encoding long-range dependencies from the previous context to allow for contextualized representations and accurate predictions. To address this challenge, we construct hierarchy of event models operating at different time-scales, predicting future events with varying granularity. This hierarchical structure enables the prediction function at any layer to extract context information dynamically from any other layer, enhancing prediction during inference (learning constraint 2). Recent approaches [42, 40, 41, 55] inspired by EST have focused on event boundary detection using predictive learning. However, these methods typically train single level and do not support higher-order predictions.
1.2 Hierarchical event models
single-level predictive model considers events that occur only at a single level of granularity rendering them unable to encode long-range, higher-order causal relationships in complex events. Conversely, a high-level representation does not contain the level of detail needed for accurately predicting low-level actions; it only encodes a high-level conceptual understanding of the action. Therefore, a hierarchy of event models is necessary to make predictions accurately at different levels of granularity [37, 26). It is necessary continuously predict future events at different levels of granularity, where low-level event models encode highly detailed information to perform short-term prediction and high-level event models encode conceptual low-detail features perform long-term prediction.
EST identifies event boundaries based transient peaks in the prediction error. To learn hierarchical structure, we extend EST: we use event models at the boundaries in a layer as inputs to the layer above. The prediction error of each layer determines event demarcation, regulating the number of inputs pooled and sent to the layer above. This enables dynamic access to long-range context for short-term prediction, required. This setup results in stacked predictive layers that perform the same prediction process with varying timescales subjective to their internal representations.
1.3 Cross-layer communication
As noted in Section 1.2, coarsely detailed long-range contexts come from higher layers (the top-left block of Figure 2), and highly detailed short-range contexts come from lower layers (the bettom-left block of Figure 2). both of which are crucial to predict future events accurately. Therefore, the prediction at each layer should be conditioned upon its own representation and those of the other layers (Equation (2)). These two types of contexts can be derived by minimizing the prediction error different layers. Hence, making perfect predictions is not the primary goal of this model but rather continuously improving its overall predictive capability.
1.3.1 Contextualized inference
A major challenge in current architectures is modeling long-range temporal dependencies between inputs. Most research has focused on modifying recurrent networks [27, 10] or extending the sequence length of transformers [11, 57] to mitigate the problem of vanishing and exploding gradients
3

---

Cross-layer communication
Top-down inference
Bottom-up optimization
Gared
Layer (1 1)
4
Egued
Layer
Layer 1)
Bottom-up inference
Top-down optimization
Figure 2: Given a stream of inputs at any layer, our model combines them and generates a bottleneck representation, which becomes the input the level above The cross-layer communication could be broken down into top-down and bottom-up contextualized inference (left) and optimization (right).
in long-range backpropagation through time [44]. Instead, we solve this problem by allowing the multi-level context representations to be shared across layers during inference. worth noting that this type of inference is rarely used in typical deep learning paradigms, where the top-down influence only comes from backpropagtting a supervised loss signal (i.e. top-down optimization). Biologically-inspiees architectures such PredNet [39] utilize top-down inference connections improve low-level predictions (i.e., frames); however, these predictive coding architectures send the prediction error signal from each low-level observation (i.e., each frame) to higher levels which prevents the network from explicitly building hierarchical levels with varying degrees of context granularity.
1.3.2 Contextualized optimization
Contextualized inference improves prediction, which is crucial for event boundary detection. However, also aim to learn rich, meaningful representations. In Section could consist of multiple interchangeable low-level events. For instance, making a sandwich can involve spreading butter or adding cheese. From a high-level, using either ingredient amounts the same parent event: "making a sandwich". Despite their visual differences, the prediction network must embed meaning and learn semantic similarities between these low-level events (i.e., spreading butter and adding cheese).
noted that 'parent' event
We implement this through "contextualized optimization" of events (Section 2.2), where each layer aligns the input representations from the lower level to minimize its own prediction loss using its context. must be noted that the contextualization from higher layers (Figure 2, bottom-right) is bal- anced by the predictive inference at the lower levels (Figure 2, top-right), which visually distinguishes the interchangeable events. This balance of optimization embeds meaningful representations into the distinct low-level representations without collapsing the model. These representations can also be utilized for event retrieval at different hierarchical levels (Figure 5). Unlike other representation learning frameworks that employ techniques like exponential moving average (EMA) or asymmetric branches prevent model collapse [7, 21, 9], we ensure that higher layers remain grounded in predicting lower-level inputs through bottom-up optimization.
2
Algorithm
Our goal is to incrementally build a stack of identical layers over the course of the learning, where each layer communicates with the layers above and below it. The layers are created needed and are trained to function different timescales; the output events from layer 7 become the inputs the layer (I + 1), as illustrated in Figure 3. We describe the model and its connections for a single layer 1,
A more detailed literature survey in the Appendix

---

Layer 2
No
Yes
33
x?2 (=.²)
(2:
Figure 3: A diagram illustrating information flow across stacked identical layers. Each layer compares its prediction Ø; with the input 2; received from the layer below. If the prediction error pped is over threshold Pr-1, the current representation žt-1 becomes the input to the layer above, and the working set is reset with æ;; otherwise, 2; is appended to the working set Xe
but the same structure applies to all the layers in the model (learning constraint 4). In what follows, describe the design mathematical model for a single predictive layer that is capable of (1) encoding temporal input into unique semantic representations (the event model) contextualized by previous events, (2) predicting the location of event boundaries (event demarcation), and (3) allowing for communication with other existing layers in the prediction stack to minimize its own prediction loss.
2.1 Temporal encoding
Let x {a> (c-m) x, 2 be set of m inputs layer I discrete time steps in the range
(I ( m, 4] where each input ar; Rdd. First, aim generate "event model"2 2(1 which is single bottleneck representation of the given inputs 20) To accomplish this, define function ((t) mxd Rd with temporally shared learnable weights (() evaluate the importance of each input in (00 for solving the prediction task hand, expressed in Equation (1).
= (x +(()))()))) (())
This event model will be trained to extract information from x'()) that is helpful for hierarchical prediction. Ideally, the bottleneck representation should encode top-down semanties, which allow for event retrieval and bottom-up interpretation of the input to minimize the prediction loss of the following input. The following subsection describes the learning objective accomplish this encoding task.
2.2 Temporal prediction
At the core of our architecture is the prediction block, which serves two purposes: event demarcation and cross-layer communication. As previously mentioned, our architecture built on the premise that minimizing the prediction loss is the only needed objective function for hierarchical event segmentation and representation learning.
Cross-layer communication allows the representation ( utilize information from higher (2(++1) ....., (44) and lower layers {2(1) ..... " (1=1))} when predicting the next input at layer 1,
We use 'event model' and 'representation' interchangeably; terminologies defined in the appendix.
5

---

where L is the total number of layers. Let 2, element the output the temporal encoding function f at its corresponding layer as expressed in Equation (1). Note that the same time variable t is used for representation across layers for simplicity; however, each layer operates in its own subjective timescale. Let ketp!) IR Lxd + R be function of 2 to predict the next input layer as expressed Equation (2)
...... = } be set of event models where each
# = (Z, L (
(2)
where ( denotes the learnable parameters of the predictor at layer 1. The difference between the layer's prediction #1+1 and the actual input = 1 is minimized, allowing the gradients to flow back into the f() functions to modify each layer's representation as expressed in Equation (3).
Speed 1+1 xt11 = (000) i+1 ~ z141
T()) "() arg min Cpeed 2 $60 (())
viE (1..))
The symbol represents an appropriate distance measure between tWO vectors.
Event demarcation is the process of detecting event boundaries by using the prediction loss, predd from Equation (3). As noted earlier, according to EST, when boundary is detected, an event model transition occurs, and a new event model is used explain the previously unpredictable observations. Instead of saving the event model to the event schemata at boundary locations as described in EST, we use it as detached input (denoted by sg(-)) to train the predictive model of the layer above (i.e., (!+1) = sg( ())). We compute the running average of the prediction loss with window of size w, expressed by Equation (5), and assume that boundary is detected when the new prediction loss is higher than the smoothed prediction loss, as expressed by the decision function in Equation (4).
{
if Cprea (át (1) ( otherwise
6( (1
(1)
P4-1
(4)
where the running average is given by
Σ i=t-w
(x) (x.))
(5)
W
2.3 Hierarchical gradient normalization
It is necessary to scale the gradients differently from conventional gradient updates because of the hierarchical nature of the model and its learning based on dynamic temporal contexts. There are three variables influencing the amount of accumulation of gradients:
1. The relative timescale between each layer is determined by the number of inputs. For instance, let the event encoder in layer (1 - 2) have seen a = (2) inputs, that at layer (l 1) have seen b = [ inputs, and that at have seen = | inputs. Then the input layer is result of seeing total of (a b c). This term can then be used to scale up the learning at any level expressed as (2.0))
2. The reach of influence of each level's representation on given level's encoder is influenced by its distance from another. For instance, if the input to f(l) comes from the levels + 2, 1,
1, 2}. then the weight of learning should be centered at / and diminish as the distance increases. Such weight at any level is given by a -[t-r] V {-2.-14 1, 0, 1, 2}. To ensure that the learning values sum up to I when this scaling is applied, the weights are normalized to add up
Eta a-|
3. The encoder receives accumulated feedback from predictors of all the layers; therefore the change prediction loss with respect to encoder parameters in any given layer should be normalized by the total number of layers, given by +.

---

The temporal encoding model can be learned by scaling its gradients expressed by the scaled Jacobian J'c( (.) in Equation (6).
Octs) C1,1 0420)
ag
C1, a4p
J'((D) C Co
(6)
2Cl4)
ac( CL,L 04 30
CL,1 (4.
where
C1,
L
2i=1 a
x
(7)
foodback
reach of inlluence
timescale
Similarly, the temporal prediction model's gradients are controlled with scaling factors as expressed in Equation (8).
J'c(D)= = Jc(()) =
Oc 8w:
ac(4) (2)
(8)
where
81
1
(9)
Σ a-1
reach of influence
3 Implementation
3.1 Training details
We resize video frames to 128 128 x 3 and use 4-layer CNN autoencoder (only for the first level) to project every frame to a single feature vector of dimension 1024 for temporal processing. For predictive-based models (STREAMER and LST,+AL), clustering- based models, we use a higher sampling rate (5 fps) to reduce noise during clustering. We cosine similarity the distance measure (~~) and use the Adam optimizer with constant learning rate of le - for training. We do not use batch normalization, regularization (i.e., dropout, weight decay), learning rate schedule, data augmentation during training. We use transformer encoder architecture for functions f and however, ablations show different architectural choices. A window size of 50 inputs (timescale respective) is used to compute the running average in Equation 5, and a new layer (I + 1) is added to the stack after layer (L) has processed 50K inputs.
sample frames fps, whereas for
3.2 Delayed gradient stepping and distributed learning
Unlike our proposed approach, conventional deep learning networks do not utilize high-level outputs in the intermediate-level predictions. Since our model includes top-down inference component, such that lower level (e.g., (Z)) backpropagates its loss gradients into the temporal encoding functions of higher level (e.g., f(>0), cannot apply the gradients immediately after loss calculation at layer (I). Therefore, we allow for scaled (i.e., Section 2.3 and Equation (8)) gradients to accumulate at all layers, then perform a single gradient step when the highest layer L backpropagates its loss.
In our streaming hierarchical learning approach, event demarcation based on the data (i.e,, some events are longer than others), posing a challenge for traditional parallclization schemes. We cannot directly batch events as inputs because each layer operates on different subjective timeline. Therefore, each model is trained separately parameters are averaged periodically during training. We train eight parallel streams on different sets of videos and average the models' parameters every 1K frames.
single stream of video data, and the models'
3.3 Datasets and comparisons
In our training and evaluation, we use two large-scale egocentric datasets: Ego4D [12] and EPIC- KITCHENS 100 [20]. Ego4D is collection of videos with total of 3670 hours of daily-life activity
7

---

Table Event segmentation comparison of MoF and average IoU, evaluated on EPIC-KITCHENS.
None of the methods listed below requires labels. The column 'Layers' refers to the number of layers evaluated against the ground truth: reports the performance of the best layer in the prediction hierarchy, whereas 3 uses the proposed Hierarchical Level Reduction algorithm for evaluation.
Protocol
Protocol 2
Method
Backbone
Pretrained
Layers
MoF
IoU
MoF t
IoU f
LSTM+AL [T]
VGGI6 [50]
ImageNet [46]
0.694
0.417
0.659
0.442
TW-FINCH [47]
0.707
0,443
0.692
0.442
Offiie ABD [16]
MTRN [60]
EPIC 50 [12]
0.704
0.438
0.699
0.432
Online ABD [16]
0.610
0.496
0.605
0.487
1
0.759
0.508
0.754
0.489
STREAMER
4-layrr CNN
0.736
0.511
0.729
0.494
collected from 74 worldwide locations. EPIC-KITCHENS 100 contains 100 hours of egocentric video, 90K action segments, and 20K unique narrations. We train our model in a self-supervised layer-by-layer manner (using only RGB frames and inputting them exactly once) on random 20% subset of Ego4D and 80% of KITCHESS then evaluate on the rest 20% EPIC-KITCHENS. We define two evaluation protocols: Protocol 1 divides EPIC-KITCHENS such that the 20% test split comes from kitchens that have not been seen in the training set, whereas Protocol 2 ensures that the kitchens in the test set are also in the training set.
We compare our method with TW-FINCH [48], Offline ABD [16], Online ABD [16], and LSTM+AL [1]. ABD, to the best of our knowledge, is the state of the art in unsupervised event segmentation. Clustering-based event segmentation models do not evaluate on egocentric datasets due to the challenges of camera motion and noise. Most clustering based- aased-aappo use pre- pre-trained or optical-flog sefical-flow-basre flow-based features, which are not effective when clustered in an egocentric setting. We re-implement ABD due to the unavailability of official code and use available official implementations for the other methods.
3.4 Evaluation metries and protocols
Event segmentation The 20K unique narrations in EPIC-KITCHENS include different labels referring the same actions (e.g., turn tap on, turn tap); therefore cannot evaluate the labeling performance of the model. We follow the protocol of LSTM+AL [1] to calculate the Jaccard index (IoU) and mean over frames (MoF) of the one-to-one mapping between the ground truth and predicted segments. Unlike LSTM+AL [1], which uses Hungarian matching to find the one-to-one mapping, we design a generalized recursive algorithm called Hierarchical Level Reduction (HLR) which finds a one-to-one mapping between the ground truth events and (a single-layer or multi-layer hierarchical) predicted events. A detailed explanation of the algorithm can be found in Supplementary Material.
Representation quality To assess the quality of the learned representations, we use the large language model (LLM) GPT 3.5 first create a dataset of events labels ranked by semantic similarity according to the L.M.. In particular, we generate KK data points sampled from EPIC-KITCHENS where each data point comprises 'query' narration and a set of 10 'key" narrations, and each key ranked by its similarity to the query. We then retrieve the features for each event in the comparison and compute the appropriate vector similarity measure, and accordingly rank each key event. This rank list is then compared with the LLM ranking to report the MSE and the Levenshtein edit distance between them. Examples of LLM similarity rankings are available in Supplementary Material.
3.5 Experiments
We evaluate STREAMER's 's performance of event segmentation and compare it with streaming and clustering- based SOTA methods as shown in Table 1. Our findings show that the performance of a single layer in STREAMER's hierarchy (the best-performing layer out of three per video) and the
Numbers in bold typeface indicate the highest performance; underline indicates the second highest
8

---

ABD
(offline)
ABD
(online)
TW-
FINCH
STREAMER
Ground
truth
Figure 4:
Qualitative comparisons of event segmentation. The Gantt chart shows more accurate alignment of STREAMER's predictions with the ground truth compared to other methods.
full 3-layer hierarchy outperform all other state of the art using IoU and MoF metrics on both testing protocols. It is worth noting that all the other methods large CNN backbone with supervised pre-trained weights (some on the same test dataset: EPIC-KITCHENS), whereas our model is trained from scratch using random initialization with simple 4-layer CNN. We show comparative qualitative results in Figure 4. More qualitative results are provided in Supplementary Material.
Additionally, evaluate the quality of event representation in Table 2. We show that self-supervising STREAMER from randomly initialized weights outperforms most clustering muusterig--based approaches with pre-trained weights; we perform par with TW-FINCH when using supervised EPIC- EPIC-KITCHEI KITCHENS pre-trained weights. Qualitative results retrieval for the first three nearest neighbors on all the events in the test split shown in Figure 5. More qualitative results reported in Supplementary Material.
Query
Method
Weights
MSE
L.
Supervised
EPIC
1.00
0.67
TW-FINCH
IN
1.018
0.710
Offline ABD
Online ABD
STREAMER
EPIC
1.02
0.71
Offline ABD
Rank
IN
1.005
0.708
EPIC
1.00
0.70
Online ABD
2
IN
1.039
0.704
No supervison
STREAMER
0.967
0.695
3
Table 2: Retrieval evaluation based on MSE and the Levenshtein edit distance (LD) of the features. All experiments are on EPIC- KITCHENS.3
Figure 5: Qualitative examples of STREAMER'S 's re- trieval of relevant events compared other methods.
Ablations We investigate three main aspects of STREAMER (Table 3): (1) varying the architecture of the temporal encoding model f. (2) varying the predictor function p. and (3) experimenting with the 'reach of influence' parameter in Equation 7. Our findings suggest that STREAMER is robust to different architectural choices of f. Our experiments also illustrate the importance of the cross-layer communication of pt simply taking the average of Z as the prediction performs worse than applying layer-specific MLP to the average: using a transformer to retrieve context from other layers dynamically performs the best. Finally, adjusting the reach of influence by gradient scaling improves the segmentation performance.
To determine the quality of the backbone features learned by STREAMER, we run ablations of using our 4-layer pretrained CNN features on SoTA clustering methods. The results, plotted in Figure 6, show significant improvement in the average mean over frames (MoF) performance of

---

event segmentation on the EPIC-KITCHENS dataset. This improvement could be attributed to the robust representations learned by the encoder through hierarchical predictive learning. particular, since these features are learned through top-down optimization, the CNN backbone is able to predict longer events higher levels, thus improving the features and contextualization quality.
MoF f
IoUt
MoF 1
IoUt
MoF T IoU1
80
GRU
LSTM
Transformer
74.03
72.8
Best HLR
0.759
0,503
0.761
0,506
0.759
0.508
70.7
70.4
0.740
0,503
0,737
0.502
0.736
0.511
MoF
Average
Average
MLP
Transformer
Best HLR
0.742
0.479
0.749
0.493
0.759
0.508
61
0.725
0.486
0.728
0.494
0.736
0.511
60
2
Best HLR
0.756
0.493
0.797
0.498
0.759
0.508
FINCH
ABD
ABD online
0.737
0.498
0.732
0.497
0.736
0.511
INMTRN/EPICO Ouur CNN/Ego4D
Table 3: Ablations studies showing the model's MoF and IoU for different values of (Equation (7)); differ- ent variants of the predictor p and the temporal encoder f. 'Best' refers the layer with the highest perfor- mance.
Figure Performance increase of SoTA clustering- based methods when using
STREAMER's pretrained 4-layer CNN features.
Conclusion
In conclusion, present STREAMER, a self-supervised and structurally evolving hierarchical temporal segmentation model that is shown to perform well on egocentric videos and is robust to hyperparameter variations. The learned representations events at different levels of granularity and semantics. As part of this, we design a gradient scaling mechanism necessary for such hierarchical frameworks with varying time-scales.
hierarchical in nature, representing
STREAMER adheres several biologically-inpiree. constraints and exhibits the ability to process long previous contexts in streaming manner, seeing each input exactly once. Our method is designed to be trained in a streaming manner which allows models to perform inference simultaneously during training, appealing to applications that require real-time adaptability [32]. We demonstrate its ability to perform event segmentation on large egocentric videos of varying perceptual conditions and demonstrate the quality of the representations through event retrieval and similarity ranking experiments.
Broader impact and limitations STREAMER requires large amounts of data to model complex high-level causal structures, and the training time increases as a layer is added. However, as self- supervised learning is becoming of increasing essence, new models must be able to continually learn from large, unlabeled data from constantly evolving domains. STREAMER caters to such online learning paradigms by fully exploiting large unlabelled video data. A much broader impact of this method extends to mudti data and domains beyond egocentric videos.
10

---

Acknowledgements
This research was supported by the US National Science Foundation Grants CNS 1513126 and IIS 1956050. The authors would like to thank Margrate Selwaness for her help with results visualizations.
References
[1] Sathyanarayanan N Aakur and Sudeep Sarkar. A perceptual prediction framework for self supervised event segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1197-1206, 2019. 8, 19
[2] Hyemin Ahn and Dongheui Lee. Refining action segmentation with hierarchical video representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16302-16310, 2021. 18
[3] Jean-Baptin baanBBapiiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste- Julien. Unsupervised learning from narrated instruction videos. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4575-4583, 2016. 18
[4] Piotr Bojanowski, Rémi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic. Weakly supervised action labeling in videos under ordering constraints. In Computer Vision-ECCC ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014. Proceedings. Part V 13, pages 628-66 643.643.666 Springer, 2014. 18
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subhiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 13:187771901,, 2020. 20
[6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 2222229999222222 2020.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sulf--uper vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9600, 2021. 4.
[8] Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. Evidence of predictive coding hierarchy in the human brain listening to speech. Nature Human Behaviour, pages 1-12. 2023. 20
[9] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-1577 2021. 20
[10] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoderr for statistical machine translation. arxiv preprint arXiv: 1078, 2014. 3
[11] Zihang Dai, Zhilin Yang. Yiming Yang. Jaime Carbonell, Quoc Le, and Ruslan Salakhutdi- nov. Transformer-xi Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901 190119200000 2019. 3
[12] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, al. Scaling egocentric vision: The epic-kitchens datasct. In Proceedings of the European Conference on Computer Vision (ECCV), pages
720-736, 2018. 7, 8
[13] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-trainine training of deep bidirec- tional transformers for language understanding, arXiv preprint 1810. 2018. 20
[14] James DiCarlo and David D Cox. Untangling invariant object recognition. Trends in cognitive sciences, 11(8):333-34 341, 2007. 3
[15] Li Ding and Chenliang Xu. Weakly-supervised action segmentation with iterative soft boundary assignment. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 6508 6516, 2018. 18
[16] Zexing Du, Xue Wang, Guoqing Zhou, and Qing Wang. Fast and unsupervised action boundary detection for action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3323-3332, 2022. 8, 18

---

[17] Yazan Abu Farha and Jurgen Gall. Ms-ten: Multi-stage temporal convolutional network for action segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pages 3575-3584, 2019. 18
[18] Karl Friston. The free-energy principle: unified brain theory? Nature reviews neuroscience, 11(2):177-
138, 2010. 3
[19] Karl Friston and Stefan Kiehel. Predictive coding under the free-energy principle. Philosophical transac- tions of the Royal Society B: Biological sciences, 364(1521):1211-122,, 2000 3
[20] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. EgoAd: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022. 7
[21] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self seffrsuperiiee learning. Advances in neural information processing systems,
33:21271 21284, 2020. 20
[22] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Proceedings of the IEEE/CVF International Conference Computer Vision Workshops, pages 0-0, O-O, 2019. 20
[23] Tengda Han, Weidi Xie, and Andrew Zisserman. Nemory-augmentt dense predictive coding for video representation learning. In Computer August 23-28, 2020, Proceedings. Part III 16, pages 312-329. Springer, 2020. 20
2020: 16th European Conference, Glasgow, UK,
[24] Jeff Hawkins and Subutai Ahmad. Why neurons have thousands of synapses, theory of sequence memory in neocortex. Frontiers in neural circuits, page 23, 2016. 2
[25] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In P Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022. 20
[26] Geoffrey Hinton. How to represent whrt- hicrarchies in a neural network. Neural Computation, pages 40, 2022. 3, 20
[27] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 175: 1997. 3
[28] Kjell Jørgen Hole and Subutai Ahmad. A thousand brains: toward biologically constrained ai. SN Applied Sciences, 3(8)7433, 2021. 2
[29] De-An Huang. Li Fei-Fei, and Juan Carlos Niebles. Connectionist temporal modeling for weakly supervised action labeling. In Computer ECsi 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 137-153. Springer, 2016.
[30] Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, and Hirokatsu Kataoka. Alleviating over- segmenntttionn errors by detecting action boundaries. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2322 2221-23312222 2021. 18
[31] Moritz Köster, Ezgi Kayhan, Miriam Langeloh, and Stefanie Hoehl. Making sense of the world: infant learning from a predictive processing perspective. Perspectives on psychological science, 15(3):562-571 2020. 2
[32] Sonu Shreshtha Kshitiz, Rany Mounir, Mayank Vatsa, Richa Singh. Saket Anand, Sudeep Sarkar, and Sevaram Mali Parihar. Long-term monitoring of bird flocks in the wild. 10
[33] Anna Kukleva, Hilde Kuehne, Fadime Sener, and Jurgen Gall. Unsupervised learning of action classes with continuous temporal embedding. In Proceedings of the IEEE/CVF Conference Computer Vision and Partern Recognition, pages 12066-12074, 2019. 19
[34] Sateesh Kumar, Sanjay Haresh, Awais Ahmed, Andrey Konin, M Zeeshan Zia, and Quoc-Huy Tran. Unsupervised action segmentation by joint representation learning and online clustering. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 20174-20188 85, 2022. 19
[35] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks for action segmentation and detection. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 356-165, 2017. 18
12

---

[36] Colin Lea, Austin Reiter, René Vidal, and Gregory D Hager. Segmental spatiotemporal cnns for fine- grained action segmentation. In Computer Vesion-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 36-52. Springer, 2016. 18
[37] Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62, 2022. 2, 20
[38] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint 1605:1605.881 2016. 20
[39] William Lotter, Gabriel Kreiman, and David D. Cox. Deep predictive coding networks for video prediction and unsupervised learning. In ICLR (Poster). OpenReview.net, 2017. 4
[40] Ramy Mounir, Sathyanarayanan Aakur, and Sudeep Sarkar. Self-supervised temporal event segmentation inspired by cognitive theories. In Advanced Methods and Deep Learning in Computer Vision, pages 405 448. Elsevier, 2022.
[41] Ramy Mounir, Roman Gula, Jöm Theuerkauf, and Sudcep Sarkar. Spatio-temporal event segmentation for wildlife extended videos. In International Conference on Computer Vision and Image Processing, pages
48-59. Springer. 2021. 3
[42] Ramy Mounir, Ahmed Shahabaz, Roman Gula, Jöm Theuerkauf, and Sudeep Sarkar. Towards auto- mated ethogramming: Cognitively-ingee -insiiee event segmentation for streaming wildlife video monitoring. International Journal of Computer Vision, pages 1-31, 2023. 3
[43] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza. Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07198, 2023. 20
[44] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318. Pmlr, 2013. 4
[45] Alexander Richard, Hilde Kuehne, and Juergen Gall. Weakly supervised action learning with mn based fine- to-coarse modeling. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 754-763, 2017. 18
[46] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, ct al. Imagenet large scale visual recognition challenge. International journul of computer vision, 115:211-2523 2015. 8
[47] Saquib Sarfraz, Naila Murray, Vivek Sharma, Ali Diba, Luc Van Gool, and Rainer Stiefelhagen. Temporally- weighted hierarchical clustering for unsupervised action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1225--12234, 2021. 8
[48] Saquib Sarfraz, Vivek Sharma, and Rainer Stiefelhagen. Efficient parameter-free clustering using first neighbor relations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 8934-8943, 2019. 8, 18
[49] Mike 7heng Shou, Stan Weixian Icc,, Weiyao Wang, Deepti Ghadiyaram, and Matt Feiszli. Generic event boundary detection: A henchmark for event segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8075-8084, 2021. 19
[50] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1.4409915566 2014. 8
[51] Yosef Singer, Yayoi Teramoto, Ben DB Willmore, Jan WH Schnupp, Andrew King, and Nicol S Harper. Sensory cortex is optimized for prediction of future input. elife, 7:c31557, 2018. 3
[52] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Du, et al. Lamda: Language models for dialog applications. arXiv preprint Xix:2201.00823333 2022. 20
[53] Trond A Tjøstheim and Andreas Stephens. Intelligence as accurate prediction. Review of Philosophy and Psychology, pages 1-25, 2021. 3
[54] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2xX1v:2302.13977, 2023. 20
13

---

[55] Xiao Wang, Jingen Liu, Tao Mei, and Jiebo Luo. Coseg: Cognitively inspired unsupervised generic event segmentation. IEEE Trunsactions Neural Networks and Learning Systems, 2023. 19
[56] Shuchen Wu, Noémi Élteto, Ishita Dasgupta, Eric Schulz. Learning structure from the ground up hierarchical representation learning by chunking. Advances in Neural Information Processing Systems, 35:36706-36721, 2022. 19
[57] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Lc. Xinet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019. 3
[58] Daniel Yon, Cecilia Heyes, and Clare Press, Beliefs and desires in the predictive brain. Nature Communi- cations, 1(1):4404, 2020. 2
[59] Jeffrey M Zacks, Nicole K Speer, Khena M Swallow, Todd S Braver, and Jeremy R Reynolds. Event perception: mind-brain perspective. Psychological bulletin, 133(2):273, 2007. 1, 19, 21
[60] Bolci Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in vidcos.
In Proceedings of the Luropean conference on computer vision (ECCV), pages 803-818, 2018.
14

---

Supplementary Material
A Hierarchical level reduction
Since the ground truth annotations are provided as a single level of event annotations, it is not possible to compare them with rich hierarchical event segmentation predicted by STREAMER. For given video and its ground truth annotations and the predicted annotations, several one-to-one mappings between them exist; desire find the with the highest average loU. In addition, it is necessary to ensure that the resulting one-to-one mapping does not contain temporally overlapping predicted annotations.
To solve this optimization problem, we design a hicrarchical level reduction (HLR) algorithm that reduces multiple layers of hierarchical events down to single layer by selecting prediction events that maximize IoU with the ground truth while ensuring no overlap of events during reduction. We design HLR recursive greedy optimization strategy. At each recursive level of Algorithm multiple ground truth events are competing to assigned to a single predicted event, so HLR returns the best of the options (line 18): (1) assigning the event the ground truth with the highest IoU, and (2) averaging the outputs of same recursive function over all children of the predicted event.
Algorithm Hierarchy Level Reduction. Given a list of the highest level annotations A'' from the predicted hierarchy and the ground truth annotations G, this algorithm finds the optimal match of the predicted annotations across the hierarchy with the ground truth while avoiding any temporal overlap between events.
Input:
A list of predicted events at the highest level L G: a list of ground truth event annotations
Output:
The overall IoU of the resulting hierarchy reduction
1: procedure FINDMATCHES(A'' ,))
2: for all g do
3: max_ious + (LoU(g, a) V VVa E A
4: a" arg max (max_ious)
5: a" www.hhhcee...mmm (rrcches.puhh( (g)
6:
a" aous. push (ious (a))
7:
end for
8: for all a E A' do
9: FINDMATCHES(a. children, a. matchs))
end for 11: end procedure
12: procedure REDUCELEVELS(a)
13:
if mmatchess = then return a. ious [0]
14:
end if
15:
h max(a. ious)
16:
chhchdidd = 0 then return h
17: else
18:
19: end max(hh,men((EEDuceLLEEvvssssss V a. m.children))
return
20:
end procedure
21: FINDMATCHES( 9)
22: return mean( (REDUURCLLLLL Vaa A4))
15

---

For models generating a hierarchical structure of events, the proposed Hierarchical Level Reduction (HLR) algorithm could be applied for comparison and evaluation. On the other hand, methods that generate single layer of event segments can directly compare to our 1-layer evaluation reported in Table
Qualitative results
This section contains qualitative results of STEAMER. A main argument of paper is that the ground truth annotations (narrations) of events in EPIC-KITCHENS are not consistent and are sometimes redundant. Figure 7 illustrates such case. Figure 8 shows an example of STREAMER s hierarchical annotations. We refer the reader to the supplementary video for more examples.
Open bag
Still opening bag
Still opening bag
Take bag out of bag
Ground truth
High
STREAMER's output
Figure 7: This figure illustrates the effect of inconsistent ground truth on the model's evaluation performance. In this segment of a video from EPIC-KITCHENS, the ground truth consists of the same narration annotated thrice succession (open bag still opening bag M, still opening bag m). Although our model could correctly detect this narration to its entirety (the middle row m), its IoU is low, thus affecting overall evaluation score. Such inconsistencies and redundancies are prevalent throughout the dataset.
Given a video snippet of an event (which we will refer to as a 'query"), can the model retrieve semantically similar video snippets from across the dataset? To determine this, perform event retrieval by representation: we first generate a representation for a random query which is then compared with the representations of events from all the videos in the dataset. Based on an appropriate similarity measure as required by the model, we select the top-few nearest matches and qualitatively examine the result.
Figure shows an example of the top-three similar matches compared with ABD. Figure 10 shows more examples of STREAMER's event retrieval, displaying the best of the top three matches. Distance in feature space calculated by the cosine similarity for our method and the Euclidean distance for ABD.
C Retrieval: quantitative analysis
In addition to the qualitative results, we perform more quantitative experiments on retrieval. As described in the main text of this work, we use the large language model (LLM) GPT 3.5 create dataset of event labels from EPIC-KITCHENS ranked by the semantic similarity. The dataset contains IK comparisons where each comparison comprises a 'query" narration and a set 10 'key" narrations, and each key is ranked by its similarity to the query according to the LLM. The keys are ranked according to the distance their representations in the feature space. The two rank lists are compared based on I) Mcan Squared Error (MSE) and 2) the Levenshtein edit distance. Listing I
16

---

High level
STREAMER'S oulput
Figure 8: Given sequence of temporal perceptual inputs (e.g., video), model learns represent them at varying levels of detail. This figure illustrates the predictions made by our model on a video from EPIC-KITCHENS three levels: the highest level (the top row in the bar chart) captures high-level, low-detail concept (scasoning vegetables ); the middle row captures events at relatively finer detail (mixing vegetables and adding salt )) and the last row captures the events in much more granular detail. Video available as supplementary.
shows the prompt used to generate the dataset and Table 5 shows some examples of LLM similarity rankings the created dataset.
prompt
Diven a list of phrase pairs, compute the semantic similarity + between the phrases in each pair and rank in the continuous range of to 10 where 10 most similar.
The list is: (queries)
Just return list decimal numbers. No exp ieplanation. """
Listing 1: The LLM prompt used to generate the dataset for retrieval quality evaluation (ranks divided by 10 to be in (0, 1))
D Implementation details
Let k be the total number of narrations in the ground truth for given video plus (for the background).
TW-FINCH, LSTM-AL on GitHub (TW-FINCH, LSTM-AL). We use the provided code to run our comparisons.
The official implementations of TW-FINCH and LSTM-AL are available
For LSTM-AL, the required number of clusters for each video being clustered was set to k. For LSTM-AL, the order of peak detection was set to 2 frames (sampled 2 fps) optimize the best results.
ABD ABD, both offline and online clustering versions, had to be re-implemented based on the implementation details of the paper. For offline clustering, window size was set to order of non-max suppression to 10, and the average number of actions to k. For the online clustering variant, the window size was set to 15, the order of non-max suppression to 40, and the lower quantile to 0.25 prescribed by the paper.
17

---

Query
Offline ABD
Online ABD
STREAMER
Rank
Rank
Rank 3
Figure 9: A comparison of top-three nearest neighbors retrievals with ABD.
E
Literature survey
In this section we provide a more detailed review related works. We start by discussing single- layered event segmentation and boundary detection works, followed by summary of existing ideas, inspirations and implementations of hierarchical models. We end our survey by relating to successful prediction-based models for representation learning.
E.1
Event segmentation
Supervised event segmentation One effective approach event segmentation is to label every frame with a class then fully supervise parameterized learning models (e.g.. neural networks) to classify each frame. This eventually groups frames into events; however, major drawback of these methods is the cost of fine-grat fine-grained frame annotations. Additionally, fully supervised methods fail to generalize well due to the labels being in a closed set. Different model variations and approaches have been tested, such as using an encoder-decoder temporal convolutional network (ED-TCN) [35], multi-stage temporal convolution network
[17], or spatiotemporal CNN model [36]. HASR [2] refines the output predictions of existing segmentation models by utilizing segment-level representations, whereas ASRF [30] improves the segmentation performances by regressing action boundary probability using representations with wide temporal receptive field.
Weakly-supervised event segmentation In order to avoid the costly process of direct frame labeling, researchers have developed weakly-supervised methods that utilize metadata (e.g., captions or narrations) to guide the training process instead of relying on explicit training labels. These approaches have been explored in various studies [45, 4, 15, 29, 3], aiming to reduce the dependency on labeled data. However, one limitation is that the required metadata may not always present in the dataset, which restricts their applicability to many real-world scenarios.
Unsupervised event segmentation To fully eliminate the need labeling, some approaches [48, 16] attempt to cluster high-level features of frames, which also climinates training. The segmentation
18

---

Query
Best match
Query
match
Random queries and the corresponding best matches, chosen from set of top-three candidates for each query on EPIC-KITCHENS.
Figure 10:
performance is directly proportional to the quality of features used in clustering. Therefore when using optical flow-based features, performance will suffer if applied to moving camera videos (e.g., egocentric) Note that in comparisons we provide these approaches with CICCKKITHENS supervised features to enhance their performance. Other methods, such as CTE [33] and TOT [34] utilize the order of actions and decode consistent labels using the Viterbi algorithm.
Event boundary detection Other works (more similar to ours) have formulated the segmentation problem as boundary detection problem. Generie Event Boundary Detection (GEBD [49]) proposed to use the difference in context embedding before and after each frame to detect boundaries as peaks. Other works [1, 55] use inspiration from [59] to detect boundaries peaks in the prediction error signal. These approaches use only low-order prediction (based on pretrained supervised high-level features) to form single layer of event representation and fail to perform higher-order predictions.
E.2 Hierarchical structure modeling
Chunking Recent work [56] provide a way to chunk data to learn a hierarchical structure by explicitly forming dictionary of patterns at each level. These methods can be challenging to scale up to real data due to the enormous number event possibilities to stored, thus demonstrated only on toy dataset examples of text and vision. Additionally, quantized inputs are required to form these
19

---

patterns. In contrast, STREAMER does not save patterns but uses learnable networks to encode and predict future inputs. We demonstrate our results on challenging egocentric data.
GLOM and H-JEPA Hinton [26] proposed an idea paper for "GLOM", which attempts to build a part-whole hierarchy by assuming cortical column for each patch in an image, where each layer in every column receives contributions from nearby columns at higher and lower levels. The hierarchy. or parse tree, is formed by forcing consensus laterally between each level, of each column, using a similarity function. Although we share the same motivation with GLOM, we significantly differ in execution. Instead using similarity function to force forming "islands of agreement",, STREAMER relies on prediction error to discover events and explicitly send their representation to higher level in a dynamically evolving structure (i.e., layer-by-layed
Perhaps we share more algorithmic similarity with LeCun's 's [37] idea paper. "H-JEPA". The paper proposes to use hierarchical joint embedding predictive architecture which uses self-supervised predictive learning to predict at different timescales at various levels in the hierarchy. This is also supported by evidence from neuroscience [8] suggesting the importance of long-range forecast representations in improving brain-mapping functionality. H-JEPA suggests that higher levels predict further into the future than lower levels; however, it does not provide a working algorithm for segmentation or cross-layer communication. It is to be noted that both works by Hinton and LeCun only provide ideas and not algorithms/results.
E.3 Predictive modeling
Natural language processing Self-supervised prediction of inputs has shown tremendous success in the field of NLP,, specifically masked language models. Recent large language models [13, 5, 5,52, 54] are based the core idea of using context to predict missing inputs. The guiding principle of training these models is that successfully predicting missing inputs implies encoding good representation of the context.
Computer vision A similar principle used to train image representation learning models. Masked Auto Encoder (MAE [25]) is trained to predict missing image patches using transformer architec- ture, thus performing better on downstream tasks using the enhanced representations. A family of representation learning methods [21, 6, 9. 7, 43] learn useful representations by predicting augmented views of the input. These augmented views attempt to simulate real-world augmentations (e.g., crop- ping simulates moving camera), therefore these methods may also be viewed predicting future frames in a video sequence. Similar ideas can also be seen applied to video representations learning. PredNet [38] attempts to implement
ceenitively-inspiiee predictive coding theories for future frame prediction. Other methods [22, 23] use dense predictive coding to learn vidco representations for retrieval and fine-tuning on downstream tasks. Unlike STREAMER, these methods do not generalize to learning hierarchical representations using predictive learning.
E.4 Relevant datasets
STREAMER is self-supervised architecture that relies on predictive learning for hierarchical segmentation. In our model, higher-order predictive layers receive sparser learning signals than lower-order layers, because the first layer directly predicts frames, whereas higher layers only receive events that cannot predicted at lower levels. Short videos not allow for higher order predictions and learning long-term temporal dependencies. Therefore, training higher levels in the hierarchy requires a large dataset (i.e., total number of hours) and longer vidcos (i.e., average video duration) in order to model high-level events. These requirements constraint the choice of datasets which we can run and evaluate STREAMER.
Based on our review of available datasets for both egocentric and exocentric settings, as shown in the Table 4, many of the available datasets, typically used in event segmentation, do not provide long enough videos. MovieNet and NewsNet are two large datasets with long videos but have not yet been released to the public. addition, MovieNet does not contain action segments; it contains coarse scene segments. The only available options to train and evaluate STREAMER is large-scale egocentric datasets, where the available datasets provide large enough scale (i.e., total number of hours) with long average duration per video for streaming and high-order temporal prediction.
20

---

Table 4: Various egocentric and exocentric datasets with total hours of recording and average duration
statistics. Not released as of this writing.
Dataset
Total Hours
Avg. Duration (min)
Large Datasets
Long Videos
Egocentric
Ego4D
3670
23
EPIC 55
10.5
EPIC 100
100
8,5
GTEA
0.58
1.23
X
X
Breakfast Action
2.3
X
YouTube Instructional
2
X
X
Exocentric
Hollywood Extended
3.7
0.23
50 Salads
FineGym
708
0.91
X
ActivityNet Captions
849
2.5
X
MovieNet
2174
117
NewsNet*
946
57
F
Glossaries
Contextualized inference In this work, contextualized inference refers the ability of the model to predict future event by using contextual representations at various levels of the event hierarchy.
Contextualized optimization Contextualized optimization refers the ability of layer to optimize the representations of other layers through its own prediction loss.
Event An event defined as segment time that is perceived by an observer to have beginning and end" [59]
Event model In cognitive psychology literature, an event model is defined as "a representa- tion of what happening now, which is robust to transient variability in the sensory input" [59]: in this work, we use 'event model" and 'representation' interchangeably.
Event demarcation, event segmentation Event demarcation is the process of detecting event boundaries by using the prediction loss, whereas event segmentation is the task of segmenting videos (or sensory inputs) into meaningful events. In other words, event segmentation the goal, and event demarcation is one way achieve it: event segmentation could be performed in other ways, such as labeling each frame in supervised framework.
Predictive learning Predictive learning refers to the brain's ability to generate predictions about future events based on past experiences.
Segmentation boundary The end of an event and the beginning of the next is segmentation boundary, marking an event transition.
21

---

Table 5: This table shows some examples of the similarity of narrations of "key events" to "query event" as determined by GPT 3.5 (text-davinci-003) A scoring means most similar, and 0 least.
Query open bottle
Keys
stop pro- cessor
open freezer
enter kitchen
open door
pick bowl
up
open fridge
clcan kitchen counter
move cheese
down
put
down
sausage
pan
0.25
0.8
0.5
0.7
0.4
0.85
0.45
0.3
0.45
0.4
open drawer
pick stone
up
check chicken
still clean-
shake off
rinse pot
pick up
close dish- washer
take pasta
out
still scoop the kiwi
pick up something
ing chop- ping board
courgette
0.75
0.55
0.65
0.65
0.7
0.8
0.7
0.85
0.65
0.8
wash leaves
wash knife
put down
rinse pan
put away
wash lid
grab hag
get kettle
pour olive oil pan 0.75
stir
pick
up
pepper 0.45
spoon
gette 0.65
forks
0.75
0.85
0.65
0.95
0.35
0.45
0.55
wash pot
wash pan
open jar
open board
pick
pick aubergine
pour milk into cereal bowl
rinse cloth
lay aubergine
close fridge
open cup- boards
22
0.85
0.65
0.75
0.8
0.85
0.9
0.85
0.8
0.75
0.75
pick up bowl
turn
off
put
down
cut lomato
stir onion
pick bowl 1.0
up
place pun
grab container 0.65
open cup- board
pour milk into glass
tap 0.75
salt
chicken 0.7
0.55
0.45
get
move
open chair washing machine door
rinse hands
check oil
pour deter-
rinse hands
check tem- perature
bread onto
wash chopping board with sponge 0.9
select schedule
chopping
gent
board
0.25
0.5
0.8
0.3
0.5
0.8
0.6
0.7
0.4
turn on tap
clean cooker
still rins- ing spatula
put colan- der on pot
put fork in bowl
put on
put lid on sun-dried tomatoes
rinse hands
wipe down counter
move
put pot
pot
mouse
0.75
0.45
0.55
0.85
0.65
0.95
0.45
0.55
0.65
0.45
open
take box
pick up jar
close
rinse mug
close hob 0.75
scrape
lift plate
wash knife
get plastic
check
drawer
drawer
tomato
trash bag 0.65
timer
0.75
0.85
0.95
0.65
0.65
0.75
0.85
0.85
close jug
shake cof- fee maker
pick plate
take kin
nap-
get weigh- ing
still taking off meat with
pick up lid
close tap
chop- ping hoard
get
put down spoon
put down howl
skin
knife 0.05
0.25
0.5
0.5
0.25
0.75
0.9
0.5
0.75
0.75
wash bowl
wash pot
put knife
put down throw
turn on tap
rinse chop-
mix nuts with oats
dry hands
place sponge away
open tap
detergent
away
ping block
onion skin
0.95
0.65
0.85
0.95
0.85
0.65
0.95
0.85


Reviews:
[{"review_id": "QyfxS0zSGe", "reviewer": "Reviewer_NKYW", "summary": "This paper presents an algorithm for unsupervised temporal segmentation of videos in an online setting, by modeling actions hierarchically. The paper is based on the idea that prediction errors arise when a new action comes up, which makes the existing context insufficient (within a certain abstraction level). When a prediction error occurs, a new hierarchical level is open that encompasses the previous event as well as the new event that is just starting.", "questions": "1. There are only 3 layers in the results. Does this mean the algorithm stops opening layers when it reaches the third level?\n\n2. At the beginning of training the first layers will be bad (untrained), so the “events” will be shorter (prediction errors will come sooner). As training evolves, layer X will have to model higher and higher level events. This can be a waste of training at the beginning, because the layers will end up modeling abstraction levels that they were not modeling initially. Have the authors considered to start training just the first layer; when it stabilizes, start training second layer (along with first layer); etc.?\n\n3. Weakness 2 is formulated as a question, so I am referencing it here.\n\n4. Are the neural networks for every layer different? Is a new neural network created for every layer? \n\n5. It would be good to have as a reference some supervised method. How far is this from a model with more direct supervision? This is in line with the general question (that would be great to get the authors' opinion on) of \"is self-supervision the answer to modeling hierarchies in videos?\" (as opposed to using some sort of supervision).\n\nOverall, the paper presents an intuitive and interesting idea. However, there are important weaknesses that need to be addressed. I believe the paper in the current form is not ready to be published, but that it can be with a couple of iterations during the rebuttal period. Therefore, expecting these weaknesses to be addressed, I recommend acceptance of the paper.\n", "limitations": "Limitations and broader impact are properly addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea in the paper is intuitive and clean. Both in terms of hierarchically representing the video, but also as a way of learning that hierarchy. Additionally, it is a good way of modeling long-term video (in terms of the potential memory problems that may arise), by encoding the past in a hierarchical way.\n\n2. The quantitative results are better than baselines.\n\n3. Good visualization videos in the supplementary. This kind of unsupervised tasks are hard to evaluate, both qualitatively and quantitatively. While the qualitative videos shown are probably cherrypicked and some times it is hard to figure out what the predicted divisions may mean, they make more sense than I would expect in such an unsupervised setting.", "weaknesses": "1. The method is not very clear from the figures or the explanation. I had to assume several things before I understood the idea, which I believe is simple and intuitive (which is good) enough to be explained in simpler terms. Because of that, I am not sure my understanding of the paper is completely correct. In the figures, where is the input video? There are two full figures showing the method and it is still not clear.\n\n2. About the idea of “changing event” when there is a prediction error: this assumption is not idea. If the prediction model is trained to be as good as possible, wouldn't it be reasonable to predict a subevent that corresponds to the next event? Is the paper assuming that that kind of prediction is impossible?\n\n3. The way the method makes predictions is (if I understand it correctly) cheating. They evaluate each level separately, and then select the level that best fits the ground truth (but they do so using ground truth). Ideally, a level should be selected *before* looking at the ground truth.\n\n4. Evaluation: the qualitative videos are good, but it would be ideal to have better explainability results or experiments. It is not clear that the method is actually outputting semantically reasonable hierarchies. For example, having human studies evaluating the hierarchies, or evaluating on hierarchical datasets (like Finegym).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your general remarks on the idea and pointing out its strengths, such as avoiding memory constraints in long-term events modeling.\n\n### Strengths:\n\n3. Thanks for this remark. We just wanted to clarify that the qualitative videos were not cherrypicked, but selected at random.\n\n### Weaknesses:\n\n1. Thank you for the suggestion, we have modified the figures to be simpler and we will include them in the camera ready version. We have included the modified figures in the pdf of the global response.\n2. Yes, we assume that perfect high-detailed long-term prediction is not possible for any of the layers, because the higher levels in the hierarchy allow for longer prediction of events by sacrificing finer details, whereas lower levels can predict high-detailed short-term events. If a well-trained layer is capable of predicting the future event given its event model, then the predicted event indeed belongs to the current event. This predicted event is however represented as a subevent in the layer below it.\n3. Our architecture outputs a rich representation of events showing a hierarchical structure; however, the groundtruth annotations only provide a single layer of events. This should be viewed as a shortcoming of the existing annotations, rather than a weakness in the output of our model. Therefore, all other unsupervised approaches have used the groundtruth in their evaluation to match their models' output to a specific segmentation granularity provided by the groundtruth. For instance, ABD [1] iteratively performs refinement on the outputs to merge neighboring events until the number of output events matches the groundtruth events, TW-FINCH [2] has a similar procedure as a clustering stopping criterion. Unlike other methods, STREAMER outputs a rich hierarchical representation, then we perform evaluation with best layer (or greedily using our proposed hierarchical layer reduction algorithm). \n4. We agree that the event segmentation quantitative results only validate the quality of boundary localization and does not explicitly validate the semantic quality of the hierarchy. As you have mentioned in the strengths (point 3), it is challenging to evaluate these kinds of unsupervised tasks; there are no agreed upon evaluation procedures/metrics for assessing the quality of a hierarchy. However, we want to point out that STREAMER restricts the output hierarchy to have aligned boundaries, the higher layers only filter out some boundaries in the lower layers based on the prediction errors. Therefore, we can deduce that if boundaries at all layers seem reasonable, the hierarchy have valid semantic relationships between layers. This kind of constraint is better than a supervised approach attempting to predict three different layers independently, as their boundaries would not align. The videos in FineGym are, on average, 50 seconds; it is therefore not possible to learn high level events with long-term temporal dependencies from such short videos. We have included a table in the global response summarizing the statistics of relevant datasets, all exhibiting the same limitation.\n\n### Questions:\n1. No, STREAMER automatically adds as many layers as needed, and this is determined by the amount of data available for training. For illustration and experiments, we show three layers, but we are not limited by three.\n2. Yes, you are correct. This is in fact how we are training STREAMER: we start by training one layer, after it has trained and stabilized, another layer is created and trained simultaneously. This is described in L28:30, L144:146, and L223:224.\n3. Please see above.\n4. Yes, a completely new layer is initialized; however, all layers have the same architecture and loss function.\n5. To the best of our knowledge, there are no supervised methods for hierarchical event segmentation on EPIC-KITCHENS datasets. In answer to the second half of the question, yes, we believe self-supervision is the solution to hierarchical modeling of events, for the following reasons:\n    - Creating hierarchical annotations manually is much more expensive than a single layer of annotations. This is clearer when considering other 2D perceptual signals, such as images. Creating multiple levels of object mask annotations (or even bounding boxes) for objects is not scalable. \n    - Hierarchical annotations can be subjective leading to inconsistent events. For example, adding salt and pepper, in a cooking video, can be represented by one person as two different high level events (with low level being grasp, shake, etc.), while a different person may consider adding salt and pepper a single event at the lowest level. Additionally, annotators may disagree on the relative level assignment between events (e.g., is \"adding salt\" at the same level as \"chopping vegetables\"?). Multiple different and semantically correct hierarchies may co-exist.\n    - We believe co-occurring patterns should be discovered using self-supervision such that discovered patterns can be used as building blocks for other higher-level patterns in a compositional structure. Forcing supervised learning on this structure can lead to poorer features that satisfy the given annotations without considering how intermediate features can be used for other higher-level tasks.\n    - It is not practical to consider all different events and patterns at all levels of the hierarchy during the manual creation of hierarchical annotations. A self-supervised approach such as STREAMER can detect all different patterns and learn semantic representations for each of them.\n\n[1] Du, Zexing, et al. \"Fast and unsupervised action boundary detection for action segmentation.\" CVPR2022.\n\n[2] Sarfraz, Saquib, et al. \"Temporally-weighted hierarchical clustering for unsupervised action segmentation.\" CVPR2021.\n\n*We hope to have addressed all your questions, and look forward to any further discussions*"}, {"review_id": "vblPQcni26", "reviewer": "Reviewer_L8go", "summary": "This paper introduces a self-supervised method to learn the hierarchical representation of streaming video events through predictive learning. This method segment videos into multiple chunks based on different semantic scale, from low-level fine-grained segmentation to high-level conceptual understanding. Meanwhile, the low-level boundaries should be consistent with the high-level ones. The authors apply predictive coding to model the causal structure of events, where the distance of the nearby video contents of variable contextual sizes decides the boundary score. In experiments, the model was trained on a random portion of the Ego4d and Epic-Kitchen datasets jointly, and it shows superior performance on the remaining video of the Epic-Kitchen dataset. ", "questions": "See weakness.", "limitations": "The paper already discussed its limitations.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The author proposed a novel mode architecture. Unlike the widely-used existing deep learning architectures, it encodes the event hierarchy in the model, while the core, prediction block, aims at both event demarcation and cross-layer communication. Therefore, the model allows the representation to utilize information from higher and lower layers to predict the next input for accurate predictive coding.\n\n2. Experiments are conducted on the union of two large-scale and challenging egocentric datasets, Ego4D and Epic-Kitchen, and the method is compared with sufficient state-of-the-art.\n\n3. The model is more lightweight than its competitors. As Table 1 shows, a 4-layer CNN autoencoder using the proposed models is already better than the state-of-the-art.", "weaknesses": "1. The paper works on a better model for understanding video streaming. However, it only conducted experiments on egocentric video datasets. It is suggested to validate the algorithm on long from exo-centric video datasets, such as MovieNet [A] and the most recent NewsNet [B].\n\n2. The proposed Hierarchical Gradient Normalization is a key component of the model optimization process. In the ablation experiment, we can see adjusting the reach of influence by gradient scaling improves the segmentation performance, but more experimental results are expected. The paper will have higher soundness when compared to different methods in this normalization process.\n\n3. The paper mentions that the federated learning method is applied during the optimization process, but the implementation is more like simple distributed learning. In federated learning, the local datasets are usually in a different data domain/distribution, which is not addressed in the paper. In addition, is it possible to enable batch-wise training on a single machine, like a modern deep-learning model usually does? \n\n\n[A] Huang, Qingqiu, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. \"Movienet: A holistic dataset for movie understanding.\" In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pp. 709-727. Springer International Publishing, 2020.\n[B] Wu, Haoqian, Keyu Chen, Haozhe Liu, Mingchen Zhuge, Bing Li, Ruizhi Qiao, Xiujun Shu et al. \"NewsNet: A Novel Dataset for Hierarchical Temporal Segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10669-10680. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n### Weaknesses:\n\n1. Thank you for the suggestion. Although NewsNet and MovieNet are large datasets with long enough videos, they have not yet been released. NewsNet was published very recently at CVPR23; it is not clear when it will be released to the public. Furthermore, in addition to the unreleased MovieNet dataset, it appears that the annotations do not contain action segments, but only coarse scene segments. We have included a table in the global response summarizing the statistics of relevant datasets.\n2. Assuming that you are suggesting to apply our hierarchical normalization process to other methods for comparison, our normalization process was designed particularly for learnable hierarchical model, which we are the first to build. Currently there are no such models on which we can apply this method for comparison. Other methods like TW-FINCH [1] use hierarchical clustering but do not have any gradient updates for normalization. Learnable methods like LSTM+AL [2] only build a single layer of event predictions, therefore they do not need hierarchical gradient normalization. That being said, our proposed normalization scheme can be applied to any future work that is designed to model hierarchical events in a streaming manner.\n3. Thank you for the remark, we will change the terminology to distributed learning. While it is possible to run on a single machine, it is not possible to enable batch-wise training due to the online event demarcation process and the streaming processing of videos. Different layers in the hierarchy receive updates at different times depending on the video contents and predictive capabilities of the model, therefore the parameters cannot be updated at the same time for all models/streams.\n\n[1] Sarfraz, Saquib, et al. \"Temporally-weighted hierarchical clustering for unsupervised action segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[2] Aakur, Sathyanarayanan N., and Sudeep Sarkar. \"A perceptual prediction framework for self supervised event segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n*We hope to have addressed all your questions, and look forward to any further discussions*"}, {"review_id": "CFbz5pYMUL", "reviewer": "Reviewer_egXC", "summary": "This paper tackles the task of event segmentation in videos, with a particular emphasis on trying to perform this task in a self-supervised setting. The authors draw inspiration from event segmentation theories from human perception to design a model, STREAMER, that semantically groups inputs at varying lengths into events (in a streaming manner), that can then be hierarchically clustered to obtain desired segmentations (evaluation is done by greedy matching high IoU overlap with groundtruth segments). The modeling approach offers a different style from standard techniques, with bottom-up and top-down optimization that encourages a natural hierarchical segmentation. The learning is also done in a streaming manner, with a single pass over the input dataset (and multiple parallel \"streams\" with a federated learning style). The authors train their proposed model on a combined subset of Ego4D and EPIC Kitchens data, and evaluate on a subset of EPIC Kitchens event segmentation (both in settings with seen and unseen kitchens).", "questions": "Overall, the preliminary rating for this paper is borderline (leaning initially positive, since I think it is good to encourage different architecture settings / reference points, and the egocentric setting is good to see as well, but overall still borderline). The key points to address for the rebuttal are in the weaknesses above. In addition, there are also a few more clarifying questions:\n1. Why does Protocol 1 have the same or better performance to Protocol 2, when Protocol 2 is the one that ensures the same kitchens from the train set are considered in the test? (per L247-248 and Table 1)\n2. What does 1 layer mean in Table 1 for this proposed method? Does this mean there is no inter-layer bottom-up/top-down message passing, as described in the technical approach? There doesn't seem to be a significant difference between 1 and 3 layers (except some tradeoff on the metrics).\n3. Also in Table 1, how does the EPIC 50 pretraining + MRTN backbone compare with that done here? (given the larger Ego4D training, is it possible that this alone accounts for differences?)\n4. If the method needs this specific egocentric setting to work (and doesn't generalize to other settings well) - what are the reasons behind such a limitation with this approach?\n\nAs an additional note, the qualitative results in the supplement are interesting, especially the cases where the EPIC annotations make better sense to merge.", "limitations": "The authors provide a discussion of limitations in the end of the paper (though I do wonder how much the layer-wise restriction applies, if they are only using 1 layer?). The societal impact seems similar to other video modeling work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "`+` The task of video event segmentation is important for video analysis and understanding in general.\n\n`+` Architecture designs inspired by human perception and event segmentation theory are generally interesting directions to push on, even if not necessarily in the mainstream.\n\n`+` The proposed technique, while it borrows aspects from different prior works on event segmentation and dense-predictive coding from self-supervised learning, feels interesting.\n\n`+` The evaluation in a challenging egocentric setting is also interesting and may provide insights for downstream applications to robotics, etc.\n\n", "weaknesses": "`-` The work seems to have a limited evaluation overall, compared to other recent work in video segmentation with SSL. For example, the key relevant prior method (ABD) performs fast unsupervised action boundary detection and they measure performance across a range of datasets from youtube instructional videos to hollywood extended (which also has a lot of background frames). In contrast, the model here focuses exclusively on the egocentric setting, with evaluation only on EPIC Kitchens, which means it is unclear how the technique generalizes to other large scale video settings.\n\n`-` Relatedly, the discussion of related work in the main paper (and to some extent, the supplement), seems to be limited regarding both the state-of-the-art in video segmentation overall (including more supervised settings with hierarchical event segmentation, like [M1]), and more classical video segmentation work (like [M2]). Having this kind of discussion would help to improve the level that the paper is self-contained, for general readers who may be less familiar with this space.\n\nexample missing references:\n[M1] \"Refining Action Segmentation with Hierarchical Video Representations\", ICCV 2021\n[M2] \"Streaming Hierarchical Video Segmentation based on an Observation Scale\", ICIP 2014\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your remarks on the qualitative results and highlighting the strengths of our idea.\n\n### Weaknesses:\n\n1. Building higher-level hierarchical events in a self-supervised manner requires long videos. Existing unsupervised methods, like ABD, are clustering-based methods which do not require training because they rely on precomputed features, therefore they can be applied to videos of any lengths. In contrast, our method learns its features from scratch. In addition, higher-order prediction tasks receive sparser learning signals than lower-order predictions (i.e., the first layer directly predicts frames, whereas higher layers only receive events that cannot be predicted at lower levels). Therefore, training higher levels in the hierarchy requires a large dataset (i.e., total number of hours) and longer videos (i.e., average video duration) in order to model high-level events (see figure to training time requirements in global response). We have included a table in the global response summarizing the statistics of relevant datasets.\n2. Thank you for the suggestions, we will included the provided references and a more detailed related work section in the camera ready submission.\n\n### Questions:\n1. These protocols help determine if the model has overfit on specific visual features (such as kitchens, lighting) or if it is capable of modeling the robust event representations instead. The performance numbers of STEAMER show that the model is learning more robust representations of events instead of learning kitchen specific actions. It seems that the model is not overfitting to specific kitchen settings, one reason could be that since the higher levels are trained to embed more robust general understanding of events (low-detail), they regularize the performance of lower level events during prediction (through top-down inference) leading to on par performance between the two evaluation protocols. Also, as you noted, some EPIC-KITCHENS groundtruth annotations are better merged, therefore, introducing inconsistencies between the protocols' test sets.\n2. Thanks for pointing this out. We will clarify it in the camera ready submission. We use two types of evaluation; “1 layer” denotes choosing a single layer out of the 3-layer hierarchy that maximizes the IoU with groudtruth annotations, whereas “3 layers” denote using the proposed Hierarchical Level Reduction algorithm (shown in the supplementary material). The small difference between both evaluation types validates that most groundtruth annotations come from a single level in our hierarchical output, which shows the consistency in our event level predictions. However, our model still outputs useful subevents and parent events to model a full hierarchy beyond what is provided in the groundtruth annotations.\n3. We do not believe that pretraining with a 4-layer CNN backbone on a subset of out-of-domain dataset (i.e., ego4D) can by itself outperform supervised features on in-domain dataset (i.e., EPIC-KITCHENS) with large backbone. The LSTM+AL [1] method has been also trained on Ego4D and we still outperform it showing the efficacy of our hierarchical approach.\n4. Our method does not have specific constraints on the type of input data it can work on; however, as mentioned in [weakness 1] above, our approach requires long ego/exo-centric for modeling higher level events. In our review of other datasets, we could not find datasets with long enough average durations outside of the egocentric domain. We have included a table in the global response summarizing the statistics of relevant datasets.\n\n### Limitation:\nJust to clarify: as mentioned in [Question 2] above, \"1-layer\" refers to evaluating the model's output by the single best layer from the hierarchy.\n\n[1] Aakur, Sathyanarayanan N., and Sudeep Sarkar. \"A perceptual prediction framework for self supervised event segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n*We hope to have addressed all your questions, and look forward to any further discussions*"}, {"review_id": "IhXtS7Mqdj", "reviewer": "Reviewer_s9dg", "summary": "This paper introduces STREAMER, a self-supervised architecture for hierarchical representation learning and segmentation of streaming perceptual inputs. The model adapts to the complexity of the input domain through layer-by-layer training, aiming to accurately predict future events while providing necessary information to other levels. By detecting prediction error peaks, an event hierarchy is constructed, enabling a bottom-up information flow. The model is fully self-supervised, trained in a streaming manner, and demonstrates promising performance on temporal event segmentation and event retrieval tasks.", "questions": "How to deal with the non-action clips in the untrimmed video sequences?\n\nHow about the computational cost of the proposed method?", "limitations": "as is mentioned in the weakness.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Most parts of this paper are well written which demonstrates its methodology and experiments. Especially, the method of this paper is easy to follow and the provided visualization is a plus to understand the proposed algorithm.\n\nThe idea of leveraging the global context is inspiring, which benefits to segment the events. In addition, the self-supervised learning pipeline significantly reduces the efforts in the annotation.\n\nThe experimental results and visualizations look promising on the Epic-Kitchen dataset.\n", "weaknesses": "The terminology used in this paper is confusing, for example, event model vs. representation, event demarcation vs. event segmentation.\n\nThe motivation of using streaming manner is not well justified in this work. It will be great to elaborate this part.\n\nIn Table 1, this work only reports the experimental results on EPIC-KITCHEN dataset, and there are few numbers of previous work comparisons. This is hard to justify the effectiveness of the proposed method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses:\n\n1.  We will clarify the terminologies in the camera ready submission. For clarification:\n    - *Event model* and *representation*: As defined in cognitive psychology literature, \"An event model is a representation of what is happening now, which is robust to transient variability in the sensory input\" [1]; as noted in Footnote 2 (Page 5), we use 'event model' and 'representation' interchangeably.\n    - *Event demarcation* and *event segmentation*. As defined in L181, \"Event demarcation is the process of detecting event boundaries by using the prediction loss\" whereas \"event segmentation\" is the task of segmenting videos (or sensory inputs) into meaning events, as is often understood in literature. Event segmentation is the goal, and event demarcation is one way to achieve it. Event segmentation could be performed in other ways, such as labelling each frame in a supervised framework.\n2. There are several motivations to learning in a streaming manner, which we will include in the camera ready submission, such as:\n    - Models trained in a streaming manner are more biologically plausible, as mentioned in L47, since the learning in the neocortex occurs continuously.\n    - Stream learning allows models to perform inference simultaneously during training which appeals to application requiring real-time adaptability\n    - When using stream learning, the model processes each input (i.e., frame) only once and does not store it to for additional training. Therefore, there are no significant storage requirements.\n    - The training can be scaled up to possibly process infinite inputs, which is not possible with conventional epoch-based training.\n    - Such setting enables learning from evolving data streams. \n3. To the best of our knowledge, we are the first to perform hierarchical event segmentation on EPIC-KITCHENS, therefore the available methods for comparison are very limited. In fact, we had to re-run/re-implement the SOTA approaches. Furthermore, building higher-level hierarchical events in a self-supervised manner requires long videos. higher-order prediction tasks receive sparser learning signals than lower-order predictions (i.e., the first layer directly predicts frames, whereas higher layers only receive events that cannot be predicted at lower levels). Therefore, training higher levels in the hierarchy requires a large dataset (i.e., total number of hours) and longer videos (i.e., average video duration) in order to model high-level events (see figure to training time requirements in global response). We have included a table in the global response summarizing the statistics of relevant datasets suggesting lack of available long-video datasets for stream hierarchical learning.\n\n### Questions:\n\n1. Our approach works on non-actions and actions clips similarly. If the event is repeatable and predictable (e.g., moving across the kitchen), then lower level event models will be able to predict all frames in the non-action clips and segment them from other action clips. Otherwise, boundaries will exist at high prediction errors.\n2. STREAMER processes frames at 3.3 frames per second (simultaneous training and inference); however, our predictive approach does not require processing consecutive frames. In practice, we sample 2 frames per second of the input 60fps videos, allowing STREAMER to process video at a speed of 90.9 fps. The results in the experimental section are shown with stream processing at 90.9 fps.\n\n[1] Jeffrey M Zacks, Nicole K Speer, Khena M Swallow, Todd S Braver, and Jeremy R Reynolds. Event perception: a mind-brain perspective. Psychological bulletin, 133(2):273, 2007\n\n*We hope to have addressed all your questions, and look forward to any further discussions*"}]

Author rebuttal:
We thank the reviewers for their comments and useful suggestions. We think that these comments have helped us better refine the paper.

A common concern expressed in the reviews is evaluation only on egocentric dataset. We address this concern here in addition to individual rebuttals.

Our method, STREAMER, is a self-supervised architecture that relies on predictive learning for hierarchical segmentation. In our model, higher-order predictive layers receive sparser learning signals than lower-order layers (see Figure 2 in the attached pdf), because the first layer directly predicts frames, whereas higher layers only receive events that cannot be predicted at lower levels. Short videos do not allow for higher order predictions and learning long-term temporal dependencies. Therefore, training higher levels in the hierarchy requires a large dataset (i.e., total number of hours) and longer videos (i.e., average video duration) in order to model high-level events. These requirements constraint the choice of datasets on which we can run and evaluate STREAMER.


| Dataset               | Egocentric    | Total Hours | Average Duration (minutes) | Large Datasets | Long Videos | Comments                                           |
|-----------------------|---------------|-------------|----------------------------|----------------|-------------|----------------------------------------------------|
| Ego4D                 | ✅            | 3670        | 23                         | ✅             | ✅          | Large dataset and long videos                      |
| EPIC-KITCHENS 55      | ✅            | 55          | 10.5                       | ✅             | ✅          | Large dataset and long videos                      |
| EPIC-KITCHENS 100     | ✅            | 100         | 8.5                        | ✅             | ✅          | Large dataset and long videos                      |
| GTEA                  | ✅            | 0.58        | 1.23                       | ❌             | ❌          | Small dataset and short videos                     |
| Breakfast Action      | ❌            | 77          | 2.3                        | ✅             | ❌          | Short videos                                       |
| YouTube Instructional | ❌            | 5           | 2                          | ❌             | ❌          | Small dataset and short videos                     |
| Hollywood Extended    | ❌            | 3.7         | 0.23                       | ❌             | ❌          | Small dataset and short videos                     |
| 50 Salads             | ❌            | 4.5         | 4.8                        | ❌             | ❌          | Small dataset and short videos                     |
| FineGym               | ❌            | 708         | 0.91                       | ✅             | ❌          | short videos                                       |
| ActivityNet Captions  | ❌            | 849         | 2.5                        | ✅             | ❌          | short videos                                       |
| MovieNet              | ❌            | 2174        | 117                        | ✅             | ✅          | Unreleased videos and contains only scene segments |
| NewsNet               | ❌            | 946         | 57                         | ✅             | ✅          | Unreleased dataset                                 |




Based on our review of available datasets for both ego-centric and exo-centric settings, as shown in the table above, we found that many of the available datasets (typically used in event segmentation) do not provide long enough videos. MovieNet and NewsNet are both large datasets with long videos but have not yet been released to the public. In addition, MovieNet does not contain action segments; it contains coarse scene segments. Whereas, the NewsNet paper was published very recently in CVPR23. The only available options to train and evaluate STREAMER is large-scale ego-centric datasets, where the available datasets provide large enough scale (i.e., total number of hours) with a long average duration per video for streaming and high-order temporal prediction. We plan to add this discussion to the supplementary paper.



Meta-review: This paper received mixed ratings: 2 borderline rejects, 1 borderline accept, and 1 weak accept. The reviewers concurred that the paper presents an intriguing and innovative idea. In particular, they found the concept of learning to represent videos in a hierarchical fashion to be intuitive, and they regarded the proposed architecture as novel. Additionally, they appreciated the generally well-written nature of the paper.

The most significant criticism, primarily from `L8go` and `s9dg`, was that the experiments were exclusively conducted on egocentric video datasets (EPIC-KITCHENS and Ego4D) and not on other "exo-centric" datasets that are perhaps more widely used in the video understanding community. Because of the limited evaluation scenario, the reviewers raised questions about the generality of the proposed approach. 

In their rebuttal, the authors argued that there are no other publicly available datasets of comparable scale and duration. They provided a table summarizing the current video datasets, emphasizing that EPIC-KITCHENS and Ego4D are only two publicly available datasets that provide both the scale and long duration.

After the rebuttal, the two reviewers `NKYW` and `egXC` concurred with the rebuttal, acknowledging that there were limited alternatives for the authors to utilize. The other two reviewers, `L8go` and `s9dg`, remained skeptical, re-iterating their core concern that the proposed approach has not been evaluated on exo-centric videos that have a wide range of applications.

During the post-rebuttal discussion phase, reviewers `NKYW` and `egXC` reiterated their recommendation to accept the paper, arguing that the paper's strengths outweighed the weakness of focusing on EPIC-KITCHENS and Ego4D for experiments. Reviewers `L8go` and `s9dg` did not participate in the discussion. This meta-reviewer found the arguments by `NKYW` and `egXC` to be persuasive. They are also well-aligned with the four reviewers' consensus that the paper presents an intriguing idea with a novel architecture. In addition, this meta-reviewer agrees with the authors that 1) the limited availability of existing exo-centric datasets (as was clearly shown in the dataset table provided in the rebuttal) makes it infeasible to evaluate on other datasets as the reviewers `L8go` and `s9dg` requested, and 2) that egocentric video understanding is equally important as exo-centric video tasks and has numerous real-world applications (which was argued in response to `L8go`). 

After careful consideration, we are happy to recommend acceptance. In the final version, we strongly recommend that the authors provide thorough explanations regarding the available large-scale, long-form video datasets (or lack thereof).
Decision: Accept (poster)