SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We resize video frames to  $128 \times 128 \times 3$  and use a 4-layer CNN autoencoder (only for the first level) to project every frame to a single feature vector of dimension 1024 for temporal processing. For predictive-based models (STREAMER and LSTM+AL), we sample frames at 2 fps, whereas for clustering-based models, we use a higher sampling rate (5 fps) to reduce noise during clustering. We use cosine similarity as the distance measure  $(\sim)$  and use the Adam optimizer with a constant learning rate of  $1e - 4$  for training. We do not use batch normalization, regularization (i.e., dropout, weight decay), learning rate schedule, or data augmentation during training. We use transformer encoder architecture for functions  $f$  and  $p$ ; however, ablations show different architectural choices. A window size  $w$  of 50 inputs (timescale respective) is used to compute the running average in Equation 5, and a new layer  $(l + 1)$  is added to the stack after layer  $(l)$  has processed 50K inputs.
3.2 Delayed gradient stepping and distributed learning
Unlike our proposed approach, conventional deep learning networks do not utilize high-level outputs in the intermediate-level predictions. Since our model includes a top-down inference component, such that a lower level  $(e.g., (l))$  backpropagates its loss gradients into the temporal encoding functions of a higher level  $(e.g., f^{(>l)})$ , we cannot apply the gradients immediately after loss calculation at layer  $(l)$ . Therefore, we allow for scaled (i.e., Section 2.3 and Equation (8)) gradients to accumulate at all layers, then perform a single gradient step when the highest layer  $L$  backpropagates its loss.
In our streaming hierarchical learning approach, event demarcation is based on the data (i.e., some events are longer than others), posing a challenge for traditional parallelization schemes. We cannot directly batch events as inputs because each layer operates on a different subjective timeline. Therefore, each model is trained separately on a single stream of video data, and the models' parameters are averaged periodically during training. We train eight parallel streams on different sets of videos and average the models' parameters every 1K frames.
3.3 Datasets and comparisons
In our training and evaluation, we use two large-scale egocentric datasets: Ego4D [12] and EPIC-KITCHENS 100 [20]. Ego4D is a collection of videos with a total of 3670 hours of daily-life activity
Table 1: Event segmentation comparison of MoF and average IoU, evaluated on EPIC-KITCHENS. None of the methods listed below requires labels. The column 'Layers' refers to the number of layers evaluated against the ground truth: 1 reports the performance of the best layer in the prediction hierarchy, whereas 3 uses the proposed Hierarchical Level Reduction algorithm for evaluation.
Method	Backbone	Pretrained	Layers	Protocol 1	Protocol 2
MoF ↑	IoU ↑	MoF ↑	IoU ↑
LSTM+AL [1]	VGG16 [50]	ImageNet [46]	1	0.694	0.417	0.659	0.442
TW-FINCH [47]	1	0.707	0.443	0.692	0.442
Offline ABD [16]	MTRN [60]	EPIC 50 [12]	1	0.704	0.438	0.699	0.432
Online ABD [16]	1	0.610	0.496	0.605	0.487
STREAMER	4-layer CNN	-	1	0.759	0.508	0.754	0.489
3	0.736	0.511	0.729	0.494
collected from 74 worldwide locations. EPIC-KITCHENS 100 contains 100 hours of egocentric video, 90K action segments, and 20K unique narrations. We train our model in a self-supervised layer-by-layer manner (using only RGB frames and inputting them exactly once) on a random  $20\%$  subset of Ego4D and  $80\%$  of EPIC-KITCHENS, then evaluate on the rest  $20\%$  of EPIC-KITCHENS. We define two evaluation protocols: Protocol 1 divides EPIC-KITCHENS such that the  $20\%$  test split comes from kitchens that have not been seen in the training set, whereas Protocol 2 ensures that the kitchens in the test set are also in the training set.
We compare our method with TW-FINCH [48], Offline ABD [16], Online ABD [16], and LSTM+AL [1]. ABD, to the best of our knowledge, is the state of the art in unsupervised event segmentation. Clustering-based event segmentation models do not evaluate on egocentric datasets due to the challenges of camera motion and noise. Most clustering based-approaches use pre-trained or optical-flow-based features, which are not effective when clustered in an egocentric setting. We re-implement ABD due to the unavailability of official code and use available official implementations for the other methods.
3.4 Evaluation metrics and protocols
Event segmentation The 20K unique narrations in EPIC-KITCHENS include different labels referring to the same actions (e.g., turn tap on, turn on tap); therefore we cannot evaluate the labeling performance of the model. We follow the protocol of LSTM+AL [1] to calculate the Jaccard index (IoU) and mean over frames (MoF) of the one-to-one mapping between the ground truth and predicted segments. Unlike LSTM+AL [1], which uses Hungarian matching to find the one-to-one mapping, we design a generalized recursive algorithm called Hierarchical Level Reduction (HLR) which finds a one-to-one mapping between the ground truth events and (a single-layer or multi-layer hierarchical) predicted events. A detailed explanation of the algorithm can be found in Supplementary Material.
Representation quality To assess the quality of the learned representations, we use the large language model (LLM) GPT 3.5 to first create a dataset of events labels ranked by semantic similarity according to the LLM. In particular, we generate 1K data points sampled from EPIC-KITCHENS, where each data point comprises a 'query' narration and a set of 10 'key' narrations, and each key is ranked by its similarity to the query. We then retrieve the features for each event in the comparison and compute the appropriate vector similarity measure, and accordingly rank each key event. This rank list is then compared with the LLM ranking to report the MSE and the Levenshtein edit distance between them. Examples of LLM similarity rankings are available in Supplementary Material.
3.5 Experiments
We evaluate STREAMER's performance of event segmentation and compare it with streaming and clustering-based SOTA methods as shown in Table 1. Our findings show that the performance of a single layer in STREAMER's hierarchy (the best-performing layer out of three per video) and the
3Numbers in bold typeface indicate the highest performance; underline indicates the second highest
Figure 4: Qualitative comparisons of event segmentation. The Gantt chart shows a more accurate alignment of STREAMER's predictions with the ground truth compared to other methods.
full 3-layer hierarchy outperform all other state of the art using IoU and MoF metrics on both testing protocols. It is worth noting that all the other methods use a large CNN backbone with supervised pre-trained weights (some on the same test dataset: EPIC-KITCHENS), whereas our model is trained from scratch using random initialization with a simple 4-layer CNN. We show comparative qualitative results in Figure 4. More qualitative results are provided in Supplementary Material.
Additionally, we evaluate the quality of event representation in Table 2. We show that self-supervising STREAMAMER from randomly initialized weights outperforms most clustering-based approaches with pre-trained weights; we perform on par with TW-FINCH when using supervised EPIC-KITCHENS pre-trained weights. Qualitative results of retrieval for the first three nearest neighbors on all the events in the test split are shown in Figure 5. More qualitative results are reported in Supplementary Material.
Method	Weights	MSE ↓	LD ↓
Supervised
TW-FINCH	EPIC	1.00	0.67
IN	1.018	0.710
Offline ABD	EPIC	1.02	0.71
IN	1.005	0.708
Online ABD	EPIC	1.00	0.70
IN	1.039	0.704
No supervision
STREAMER	-	0.967	0.695
Table 2: Retrieval evaluation based on MSE and the Levenshtein edit distance (LD) of the features. All experiments are on EPIC-KITCHENS.3
Figure 5: Qualitative examples of STREAMAMER's retrieval of relevant events compared to other methods.
Ablations We investigate three main aspects of STREAMAMER (Table 3): (1) varying the architecture of the temporal encoding model  $f$ , (2) varying the predictor function  $p$ , and (3) experimenting with the 'reach of influence' parameter  $\alpha$  in Equation 7. Our findings suggest that STREAMAMER is robust to different architectural choices of  $f$ . Our experiments also illustrate the importance of the cross-layer communication of  $p$ : simply taking the average of  $\mathcal{Z}$  as the prediction performs worse than applying a layer-specific MLP to the average; using a transformer to retrieve context from other layers dynamically performs the best. Finally, adjusting the reach of influence by gradient scaling improves the segmentation performance.
To determine the quality of the backbone features learned by STREAMAMER, we run ablations of using our 4-layer pretrained CNN features on SoTA clustering methods. The results, plotted in Figure 6, show significant improvement in the average mean over frames (MoF) performance of
event segmentation on the EPIC-KITCHENS dataset. This improvement could be attributed to the robust representations learned by the encoder through hierarchical predictive learning. In particular, since these features are learned through top-down optimization, the CNN backbone is able to predict longer events at higher levels, thus improving the features and contextualization quality.
MoF ↑	IoU ↑	MoF ↑	IoU ↑	MoF ↑	IoU ↑
f	GRU	LSTM	Transformer
Best	0.759	0.503	0.761	0.506	0.759	0.508
HLR	0.740	0.503	0.737	0.502	0.736	0.511
p	Average	Average + MLP	Transformer
Best	0.742	0.479	0.749	0.493	0.759	0.508
HLR	0.725	0.486	0.728	0.494	0.736	0.511
α	1	2	3
Best	0.756	0.493	0.797	0.498	0.759	0.508
HLR	0.737	0.498	0.732	0.497	0.736	0.511
Table 3: Ablations studies showing the model's MoF and IoU for different values of  $\alpha$  (Equation (7)); different variants of the predictor  $p$  and the temporal encoder  $f$ . 'Best' refers to the layer with the highest performance.
Figure 6: Performance increase of SoTA clustering-based methods when using STREAMAMER's pretrained 4-layer CNN features.
In conclusion, we present STREAMAMER, a self-supervised and structurally evolving hierarchical temporal segmentation model that is shown to perform well on egocentric videos and is robust to hyperparameter variations. The learned representations are hierarchical in nature, representing events at different levels of granularity and semantics. As part of this, we design a gradient scaling mechanism necessary for such hierarchical frameworks with varying time-scales.
STREAMAMER adheres to several biologically-inspired constraints and exhibits the ability to process long previous contexts in a streaming manner, seeing each input exactly once. Our method is designed to be trained in a streaming manner which allows models to perform inference simultaneously during training, appealing to applications that require real-time adaptability [32]. We demonstrate its ability to perform event segmentation on large egocentric videos of varying perceptual conditions and demonstrate the quality of the representations through event retrieval and similarity ranking experiments.
Broader impact and limitations STREAMER requires large amounts of data to model complex high-level causal structures, and the training time increases as a layer is added. However, as self-supervised learning is becoming of increasing essence, new models must be able to continually learn from large, unlabeled data from constantly evolving domains. STREAMER caters to such online learning paradigms by fully exploiting large unlabeled video data. A much broader impact of this method extends to multi-modal data and domains beyond egocentric videos.
This research was supported by the US National Science Foundation Grants CNS 1513126 and IIS 1956050. The authors would like to thank Margrate Selwaness for her help with results visualizations.
[55] Xiao Wang, Jingen Liu, Tao Mei, and Jiebo Luo. Coseg: Cognitively inspired unsupervised generic event segmentation. IEEE Transactions on Neural Networks and Learning Systems, 2023. 3, 19
[56] Shuchen Wu, Noémi Élteto, Ishita Dasgupta, and Eric Schulz. Learning structure from the ground up— hierarchical representation learning by chunking. Advances in Neural Information Processing Systems, 35:36706-36721, 2022. 19
[57] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019. 3
[58] Daniel Yon, Cecilia Heyes, and Clare Press. Beliefs and desires in the predictive brain. Nature Communications, 11(1):4404, 2020. 2
[59] Jeffrey M Zacks, Nicole K Speer, Khena M Swallow, Todd S Braver, and Jeremy R Reynolds. Event perception: a mind-brain perspective. *Psychological bulletin*, 133(2):273, 2007. 1, 3, 19, 21
[60] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European conference on computer vision (ECCV), pages 803-818, 2018. 8
A Hierarchical level reduction
Since the ground truth annotations are provided as a single level of event annotations, it is not possible to compare them with rich hierarchical event segmentation predicted by STREAMAMER. For a given video and its ground truth annotations and the predicted annotations, several one-to-one mappings between them exist; we desire to find the one with the highest average IoU. In addition, it is necessary to ensure that the resulting one-to-one mapping does not contain temporally overlapping predicted annotations.
To solve this optimization problem, we design a hierarchical level reduction (HLR) algorithm that reduces multiple layers of hierarchical events down to a single layer by selecting prediction events that maximize IoU with the ground truth while ensuring no overlap of events during reduction. We design HLR as a recursive greedy optimization strategy. At each recursive level of Algorithm 1, multiple ground truth events are competing to be assigned to a single predicted event, so HLR returns the best of the two options (line 18): (1) assigning the event to the ground truth with the highest IoU, and (2) averaging the outputs of same recursive function over all children of the predicted event.
Algorithm 1: Hierarchy Level Reduction. Given a list of the highest level annotations  $\mathcal{A}^L$  from the predicted hierarchy and the ground truth annotations  $\mathcal{G}$ , this algorithm finds the optimal match of the predicted annotations across the hierarchy with the ground truth while avoiding any temporal overlap between events.
$\mathcal{A}^L$ : a list of predicted events at the highest level  $L$
$\mathcal{G}$  : a list of ground truth event annotations
The overall IoU of the resulting hierarchy reduction
1: procedure FINDMATCHES  $(\mathcal{A}^l,\mathcal{G})$
2: for all  $g\in \mathcal{G}$  do
3: max_ious  $\leftarrow \{\mathrm{IoU}(\pmb {g},\pmb {a})\mid \forall \pmb {a}\in \mathcal{A}^l\}$
4:  $\mathbf{a}^{*} \gets \arg \max \left( \max_{\text{ious}} \right)$
5:  $a^{*}$  .matches.push  $(g)$
6:  $a^*$  .ious.push(ious[a\*])
7: end for
8: for all  $\pmb{a} \in \mathcal{A}^l$  do
9: FINDMATCHES(a children, a matches)
10: end for
11: end procedure
12: procedure REDUCELEVELS(a)
13: if  $|a$ .matches| = 1 then return a.ious[0]
14: end if
15:  $h\gets \max (\pmb {a}.$  ious)
16: if  $|a$ .children| = 0 then return h
17: else
18: return max(h, mean({REDUCELEVELS(c)) |  $\forall c\in a$  .children})
19: end if
20: end procedure
21: FINDMATCHES  $(\mathcal{A}^L,\mathcal{G})$
22: return mean(\{REDUCELEVELS  $(\pmb{a})$ $|\forall \pmb {a}\in \mathcal{A}^L\}$
For models generating a hierarchical structure of events, the proposed Hierarchical Level Reduction (HLR) algorithm could be applied for comparison and evaluation. On the other hand, methods that generate a single layer of event segments can directly compare to our 1-layer evaluation reported in Table 1.
B Qualitative results
This section contains more qualitative results of STEAMER. A main argument of our paper is that the ground truth annotations (narrations) of events in EPIC-KITCHENS are not consistent and are sometimes redundant. Figure 7 illustrates one such case. Figure 8 shows an example of STREAMAMER's hierarchical annotations. We refer the reader to the supplementary video for more examples.
Figure 7: This figure illustrates the effect of inconsistent ground truth on the model's evaluation performance. In this segment of a video from EPIC-KITCHENS, the ground truth consists of the same narration annotated thrice in succession (open bag  $\square$ , still opening bag  $\square$ , still opening bag  $\square$ ). Although our model could correctly detect this narration to its entirety (the middle row  $\square$ ), its IoU is low, thus affecting its overall evaluation score. Such inconsistencies and redundancies are prevalent throughout the dataset.
Given a video snippet of an event (which we will refer to as a 'query'), can the model retrieve semantically similar video snippets from across the dataset? To determine this, we perform event retrieval by representation: we first generate a representation for a random query which is then compared with the representations of events from all the videos in the dataset. Based on an appropriate similarity measure as required by the model, we select the top-few nearest matches and qualitatively examine the result.
Figure 9 shows an example of the top-three similar matches compared with ABD. Figure 10 shows more examples of STREAMER's event retrieval, displaying the best of the top three matches. Distance in feature space is calculated by the cosine similarity for our method and the Euclidean distance for ABD.
C Retrieval: quantitative analysis
In addition to the qualitative results, we perform more quantitative experiments on retrieval. As described in the main text of this work, we use the large language model (LLM) GPT 3.5 to create a dataset of event labels from EPIC-KITCHENS ranked by the semantic similarity. The dataset contains 1K comparisons where each comparison comprises a 'query' narration and a set of 10 'key' narrations, and each key is ranked by its similarity to the query according to the LLM. The keys are ranked according to the distance of their representations in the feature space. The two rank lists are compared based on 1) Mean Squared Error (MSE) and 2) the Levenshtein edit distance. Listing 1
Figure 8: Given a sequence of temporal perceptual inputs (e.g., video), our model learns to represent them at varying levels of detail. This figure illustrates the predictions made by our model on a video from EPIC-KITCHENS at three levels: the highest level (the top row in the bar chart) captures a high-level, low-detail concept (seasoning vegetables  $\blacksquare$ ); the middle row captures events at relatively finer detail (mixing vegetables  $\blacksquare$  and adding salt  $\blacksquare$ ); and the last row captures the events in much more granular detail. Video available as supplementary.
shows the prompt used to generate the dataset and Table 5 shows some examples of LLM similarity rankings in the created dataset.
prompt  $=$  f""  
Given a list of phrase pairs, compute the semantic similarity  $\leftrightarrow$  between the phrases in each pair and rank in the continuous range of 0 to 10 where 10 is most similar.  
The list is: {queries}  
Just return a list of decimal numbers. No explanation.\n"''"
Listing 1: The LLM prompt used to generate the dataset for retrieval quality evaluation (ranks divided by 10 to be in  $(0,1)$ )
Let  $k$  be the total number of narrations in the ground truth for a given video plus one (for the background).
TW-FINCH, LSTM-AL The official implementations of TW-FINCH and LSTM-AL are available on GitHub (TW-FINCH, LSTM-AL). We use the provided code to run our comparisons.
For LSTM-AL, the required number of clusters for each video being clustered was set to  $k$ . For LSTM-AL, the order of peak detection was set to 2 frames (sampled at 2 fps) to optimize the best results.
ABD ABD, both offline and online clustering versions, had to be re-implemented based on the implementation details of the paper. For offline clustering, the window size was set to 5, the order of non-max suppression to 10, and the average number of actions to  $k$ . For the online clustering variant, the window size was set to 15, the order of non-max suppression to 40, and the lower quantile to 0.25 as prescribed by the paper.
Figure 9: A comparison of top-three nearest neighbors retrievals with ABD.
E Literature survey
In this section we provide a more detailed review of related works. We start by discussing single-layered event segmentation and boundary detection works, followed by a summary of existing ideas, inspirations and implementations of hierarchical models. We end our survey by relating to successful prediction-based models for representation learning.
E.1 Event segmentation
Supervised event segmentation One effective approach to event segmentation is to label every frame with a class then fully supervise parameterized learning models (e.g., neural networks) to classify each frame. This eventually groups frames into events; however, a major drawback of these methods is the cost of fine-grained frame annotations. Additionally, fully supervised methods fail to generalize well due to the labels being in a closed set. Different model variations and approaches have been tested, such as using an encoder-decoder temporal convolutional network (ED-TCN) [35], multi-stage temporal convolution network (MS-TCN) [17], or a spatiotemporal CNN model [36]. HASR [2] refines the output predictions of existing segmentation models by utilizing segment-level representations, whereas ASRF [30] improves the segmentation performances by regressing the action boundary probability using representations with a wide temporal receptive field.
Weakly-supervised event segmentation In order to avoid the costly process of direct frame labeling, researchers have developed weakly-supervised methods that utilize metadata (e.g., captions or narrations) to guide the training process instead of relying on explicit training labels. These approaches have been explored in various studies [45, 4, 15, 29, 3], aiming to reduce the dependency on labeled data. However, one limitation is that the required metadata may not always present in the dataset, which restricts their applicability to many real-world scenarios.
Unsupervised event segmentation To fully eliminate the need for labeling, some approaches [48, 16] attempt to cluster high-level features of frames, which also eliminates training. The segmentation
Figure 10: Random queries and the corresponding best matches, chosen from a set of top-three candidates for each query on EPIC-KITCHENS.
performance is directly proportional to the quality of features used in clustering. Therefore when using optical flow-based features, performance will suffer if applied to moving camera videos (e.g., egocentric). Note that in our comparisons we provide these approaches with EPIC-KITCHENS supervised features to enhance their performance. Other methods, such as CTE [33] and TOT [34] utilize the order of actions and decode consistent labels using the Viterbi algorithm.
Event boundary detection Other works (more similar to ours) have formulated the segmentation problem as a boundary detection problem. Generic Event Boundary Detection (GEBD [49]) proposed to use the difference in context embedding before and after each frame to detect boundaries as peaks. Other works [1, 55] use inspiration from [59] to detect boundaries as peaks in the prediction error signal. These approaches use only low-order prediction (based on pretrained supervised high-level features) to form a single layer of event representation and fail to perform higher-order predictions.
E.2 Hierarchical structure modeling