SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ramy Mounir Sujal Vijayaraghavan Sudeep Sarkar  
Department of Computer Science and Engineering, University of South Florida, Tampa  
{ramy, sujal, sarkar}@usf.edu
We present a novel self-supervised approach for hierarchical representation learning and segmentation of perceptual inputs in a streaming fashion. Our research addresses how to semantically group streaming inputs into chunks at various levels of a hierarchy while simultaneously learning, for each chunk, robust global representations throughout the domain. To achieve this, we propose STREAMAMER, an architecture that is trained layer-by-layer, adapting to the complexity of the input domain. In our approach, each layer is trained with two primary objectives: making accurate predictions into the future and providing necessary information to other levels for achieving the same objective. The event hierarchy is constructed by detecting prediction error peaks at different levels, where a detected boundary triggers a bottom-up information flow. At an event boundary, the encoded representation of inputs at one layer becomes the input to a higher-level layer. Additionally, we design a communication module that facilitates top-down and bottom-up exchange of information during the prediction process. Notably, our model is fully self-supervised and trained in a streaming manner, enabling a single pass on the training data. This means that the model encounters each input only once and does not store the data. We evaluate the performance of our model on the egocentric EPIC-KITCHENS dataset, specifically focusing on temporal event segmentation. Furthermore, we conduct event retrieval experiments using the learned representations to demonstrate the high quality of our video event representations. Illustration videos and code are available on our project page: https://ramymounir.com/publications/streamer.
1 Computational theory
In temporal event analysis, an event is defined as "a segment in time that is perceived by an observer to have a beginning and an end" [59]. Events could be described by a sequence of constituent events of relatively finer detail, thus forming a hierarchical structure. The end of an event and the beginning of the next is a segmentation boundary, marking an event transition. Segmentation boundaries in the lower levels of the hierarchy represent event transitions at relatively granular scales, whereas boundaries in higher levels denote higher-level event transitions.
We propose a structurally self-evolving model to learn the hierarchical representation of such events in a self-supervised streaming fashion through predictive learning. Structural evolution refers to the model's capability to create learnable layers ad hoc during training. One may argue that existing deep learning architectures are compositional in nature, where high-level features are composed of lower-level features, forming a hierarchy of features. However, it is important to distinguish between a feature hierarchy and an event hierarchy: an event hierarchy is similar to a part/whole hierarchy in the sense that each event has clear boundaries that reflect the beginning and the end of a coherent chunk of information. One may also view the hierarchy as a redundancy pooling mechanism, where
Figure 1: Comparison of STREAMAMER's hierarchical output to single-level ground truth annotations from EPIC-KITCHENS. The ground truth contains redundant narrations for successive annotations (e.g., add chicken  $\square$ ,  $\square$ ); STREAMAMER identifies such instances as a single high level event  $(\square)$ . (Narrations from ground truth)
information grouped as one event is considered redundant for a higher level and can be summarized into a single representation for higher-level processing.
Our model is capable of generating a hierarchy of event segments (Figure 1) by learning unique semantic representations for each event type directly from video frames. This is achieved through predictive learning, which models the causal structure of events. These learned representations are expressive enough to enable video snippet retrieval across videos. Each level in the hierarchy selectively groups inputs from the level below to form coherent event representations, which are then sent to the level above. As a result, the hierarchy exhibits temporally aligned boundaries, with each level containing a subset of the boundaries detected in the lower level.
As often prescribed [28, 24], we impose the following biologically-plausible constraints on our learning algorithm:
1. The learning algorithm should be continuous and online. Most existing learning algorithms offer batch-based offline learning. However, the learning in the neocortex occurs continuously in a streaming fashion while seeing each datapoint only once
2. The learning should involve the ability to make high-order predictions by "incorporating contextual information from the past. The network needs to dynamically determine how much temporal context is needed to make the best predictions" [24] (Section 1.1)
3. Learning algorithms should be self-supervised and should not assume labels for training [37]; instead, they should be able to figure out the learning objective from patterns and causal structures within the data
4. The learning should stem from a universal general-purpose algorithm. This is supported by observations of the brain circuitry showing that all neocortical regions are doing the same task in a repeated structure of cells [24]. Therefore, there should be no need for a global loss function (i.e., end-to-end training with high-level labels); local learning rules should suffice (Section 2)
1.1 Predictive learning
Predictive learning refers to the brain's ability to generate predictions about future events based on past experiences. It is a fundamental process in human cognition that guides perception, action, and thought [58, 31]. The discrepancy between the brain's predictions and the observed perceptual
inputs forms a useful training signal for optimizing cortical functions: if a model can predict into the future, it implies that it has learned the underlying causal structure of the surrounding environment. Theories of cognition hypothesize that the brain only extracts and selects features from the previous context that help in minimizing future prediction errors, thus making the sensory cortex optimized for prediction of future input [51]. A measure of intelligence can be formulated as the ability of a model to generate accurate, long-range future prediction [53]. To this end, we design an architecture with the main goal of minimizing the prediction error, also referred to as maximizing the model evidence in Bayesian inference according to the free energy principle [19, 18].
Event segmentation theory (EST) suggests that desirable properties such as event segmentation emerge as a byproduct of minimizing the prediction loss [59]. Humans are capable of chunking streaming perceptual inputs into events (and chunking spatial regions into objects [14]) to allow for memory consolidation and event retrieval for better future predictions. EST breaks down streaming sensory input into chunks by detecting event boundaries as transient peaks in the prediction error. The detected boundaries trigger a process of transitioning (i.e., shifting) to a new event model whereby the current event model is saved in the event schemata, and a different event model is retrieved, or a new one initialized to better explain the new observations. One challenge in implementing a computational model of EST is encoding long-range dependencies from the previous context to allow for contextualized representations and accurate predictions. To address this challenge, we construct a hierarchy of event models operating at different time-scales, predicting future events with varying granularity. This hierarchical structure enables the prediction function at any layer to extract context information dynamically from any other layer, enhancing prediction during inference (learning constraint 2). Recent approaches [42, 40, 41, 55] inspired by EST have focused on event boundary detection using predictive learning. However, these methods typically train a single level and do not support higher-order predictions.
1.2 Hierarchical event models
A single-level predictive model considers events that occur only at a single level of granularity rendering them unable to encode long-range, higher-order causal relationships in complex events. Conversely, a high-level representation does not contain the level of detail needed for accurately predicting low-level actions; it only encodes a high-level conceptual understanding of the action. Therefore, a hierarchy of event models is necessary to make predictions accurately at different levels of granularity [37, 26]. It is necessary to continuously predict future events at different levels of granularity, where low-level event models encode highly detailed information to perform short-term prediction and high-level event models encode conceptual low-detail features to perform long-term prediction.
EST identifies event boundaries based on transient peaks in the prediction error. To learn a hierarchical structure, we extend EST: we use event models at the boundaries in a layer as inputs to the layer above. The prediction error of each layer determines event demarcation, regulating the number of inputs pooled and sent to the layer above. This enables dynamic access to long-range context for short-term prediction, as required. This setup results in stacked predictive layers that perform the same prediction process with varying timescales subjective to their internal representations.
1.3 Cross-layer communication
As noted in Section 1.2, coarsely detailed long-range contexts come from higher layers (the top-left block of Figure 2), and highly detailed short-range contexts come from lower layers (the bottom-left block of Figure 2), both of which are crucial to predict future events accurately. Therefore, the prediction at each layer should be conditioned upon its own representation and those of the other layers (Equation (2)). These two types of contexts can be derived by minimizing the prediction error at different layers. Hence, making perfect predictions is not the primary goal of this model but rather continuously improving its overall predictive capability.
1.3.1 Contextualized inference
A major challenge in current architectures is modeling long-range temporal dependencies between inputs. Most research has focused on modifying recurrent networks [27, 10] or extending the sequence length of transformers [11, 57] to mitigate the problem of vanishing and exploding gradients
Figure 2: Given a stream of inputs at any layer, our model combines them and generates a bottleneck representation, which becomes the input to the level above it. The cross-layer communication could be broken down into top-down and bottom-up contextualized inference (left) and optimization (right).
in long-range backpropagation through time [44]. Instead, we solve this problem by allowing the multi-level context representations to be shared across layers during inference. It is worth noting that this type of inference is rarely used in typical deep learning paradigms, where the top-down influence only comes from backpropagating a supervised loss signal (i.e., top-down optimization). Biologically-inspired architectures such as PredNet [39] utilize top-down inference connections to improve low-level predictions (i.e., frames); however, these predictive coding architectures send the prediction error signal from each low-level observation (i.e., each frame) to higher levels which prevents the network from explicitly building hierarchical levels with varying degrees of context granularity.
1.3.2 Contextualized optimization
Contextualized inference improves prediction, which is crucial for event boundary detection. However, we also aim to learn rich, meaningful representations. In Section 1, we noted that a 'parent' event could consist of multiple interchangeable low-level events. For instance, making a sandwich can involve spreading butter or adding cheese. From a high-level, using either ingredient amounts to the same parent event: "making a sandwich". Despite their visual differences, the prediction network must embed meaning and learn semantic similarities between these low-level events (i.e., spreading butter and adding cheese).
We implement this through "contextualized optimization" of events (Section 2.2), where each layer aligns the input representations from the lower level to minimize its own prediction loss using its context. It must be noted that the contextualization from higher layers (Figure 2, bottom-right) is balanced by the predictive inference at the lower levels (Figure 2, top-right), which visually distinguishes the interchangeable events. This balance of optimization embeds meaningful representations into the distinct low-level representations without collapsing the model. These representations can also be utilized for event retrieval at different hierarchical levels (Figure 5). Unlike other representation learning frameworks that employ techniques like exponential moving average (EMA) or asymmetric branches to prevent model collapse [7, 21, 9], we ensure that higher layers remain grounded in predicting lower-level inputs through bottom-up optimization.
Our goal is to incrementally build a stack of identical layers over the course of the learning, where each layer communicates with the layers above and below it. The layers are created as needed and are trained to function at different timescales; the output events from layer  $l$  become the inputs to the layer  $(l + 1)$ , as illustrated in Figure 3. We describe the model and its connections for a single layer  $l$ ,
A more detailed literature survey is in the Appendix
Figure 3: A diagram illustrating information flow across stacked identical layers. Each layer compares its prediction  $\hat{\pmb{x}}_t$  with the input  $\pmb{x}_t$  received from the layer below. If the prediction error  $\mathcal{L}_{\mathrm{pred}}$  is over a threshold  $\mu_{t - 1}$ , the current representation  $\pmb{z}_{t - 1}$  becomes the input to the layer above, and the working set is reset with  $\pmb{x}_t$ ; otherwise,  $\pmb{x}_t$  is appended to the working set  $\mathcal{X}_t$
but the same structure applies to all the layers in the model (learning constraint 4). In what follows, we describe the design of a mathematical model for a single predictive layer that is capable of (1) encoding temporal input into unique semantic representations (the event model) contextualized by previous events, (2) predicting the location of event boundaries (event demarcation), and (3) allowing for communication with other existing layers in the prediction stack to minimize its own prediction loss.
2.1 Temporal encoding
Let  $\mathcal{X}^{(l)} = \{\pmb{x}_{(t - m)}^{(l)},\dots ,\pmb{x}_t^{(l)}\}$  be a set of  $m$  inputs to a layer  $l$  at discrete time steps in the range  $(t - m,t]$  where each input  $\pmb {x}_i\in \mathbb{R}^d$ . First, we aim to generate an "event model"2  $\pmb{z}^{(l)}$  which is a single bottleneck representation of the given inputs  $\mathcal{X}^{(l)}$ . To accomplish this, we define a function  $f^{(l)}:\mathbb{R}^{m\times d}\mapsto \mathbb{R}^d$  with temporally shared learnable weights  $\Phi^{(l)}$  to evaluate the importance of each input in  $\mathcal{X}^{(l)}$  for solving the prediction task at hand, as expressed in Equation (1).
$$
\boldsymbol {z} ^ {(l)} = f ^ {(l)} \left(\mathcal {X} ^ {(l)}; \boldsymbol {\Phi} ^ {(l)}\right) \tag {1}
$$
This event model will be trained to extract information from  $\mathcal{X}^{(l)}$  that is helpful for hierarchical prediction. Ideally, the bottleneck representation should encode top-down semantics, which allow for event retrieval and a bottom-up interpretation of the input to minimize the prediction loss of the following input. The following subsection describes the learning objective to accomplish this encoding task.
2.2 Temporal prediction
At the core of our architecture is the prediction block, which serves two purposes: event demarcation and cross-layer communication. As previously mentioned, our architecture is built on the premise that minimizing the prediction loss is the only needed objective function for hierarchical event segmentation and representation learning.
Cross-layer communication allows the representation  $\mathbf{z}^{(l)}$  to utilize information from higher  $\{z^{(l + 1)},\ldots ,z^{(L)}\}$  and lower layers  $\{z^{(1)},\dots ,z^{(l - 1)}\}$  when predicting the next input at layer  $l$ ,
2We use 'event model' and 'representation' interchangeably; terminologies defined in the appendix.
where  $L$  is the total number of layers. Let  $\mathcal{Z}_t = \{\pmb{z}_t^{(1)},\dots,\pmb{z}_t^{(L)}\}$  be a set of event models where each element is the output of the temporal encoding function  $f$  at its corresponding layer as expressed in Equation (1). Note that the same time variable  $t$  is used for representation across layers for simplicity; however, each layer operates in its own subjective timescale. Let  $p^{(l)}:\mathbb{R}^{L\times d}\mapsto \mathbb{R}^d$  be a function of  $\mathcal{Z}$  to predict the next input at layer  $l$  as expressed in Equation (2)
$$
\hat {\boldsymbol {x}} _ {t + 1} ^ {(l)} = p ^ {(l)} \left(\mathcal {Z} _ {t}; \Psi^ {(l)}\right) \tag {2}
$$
where  $\Psi^{(l)}$  denotes the learnable parameters of the predictor at layer  $l$ . The difference between the layer's prediction  $\hat{\pmb{x}}_{t + 1}^{(l)}$  and the actual input  $\pmb{x}_{t + 1}^{(l)}$  is minimized, allowing the gradients to flow back into the  $f^{(\cdot)}$  functions to modify each layer's representation as expressed in Equation (3).
$$
\begin{array}{l} \mathcal {L} _ {\text {p r e d}} \left(\hat {\boldsymbol {x}} _ {t + 1} ^ {(l)}, \boldsymbol {x} _ {t + 1} ^ {(l)}\right) = \hat {\boldsymbol {x}} _ {t + 1} ^ {(l)} \sim \boldsymbol {x} _ {t + 1} ^ {(l)} \\ \boldsymbol {\Phi} ^ {* (i)}, \boldsymbol {\Psi} ^ {* (l)} \leftarrow \underset {\boldsymbol {\Phi} ^ {(i)}, \boldsymbol {\Psi} ^ {(l)}} {\arg \min } \mathcal {L} _ {\operatorname {p r e d}} \left(\hat {\boldsymbol {x}} _ {t + 1} ^ {(l)}, \boldsymbol {x} _ {t + 1} ^ {(l)}\right) \quad \forall i \in \{1 \dots L \} \tag {3} \\ \end{array}
$$
The symbol  $\sim$  represents an appropriate distance measure between two vectors.
Event demarcation is the process of detecting event boundaries by using the prediction loss,  $\mathcal{L}_{\mathrm{pred}}$ , from Equation (3). As noted earlier, according to EST, when a boundary is detected, an event model transition occurs, and a new event model is used to explain the previously unpredictable observations. Instead of saving the event model to the event schemata at boundary locations as described in EST, we use it as a detached input (denoted by  $\operatorname{sg}[\cdot]$ ) to train the predictive model of the layer above it (i.e.,  $\pmb{x}^{(l+1)} \equiv \operatorname{sg}[\pmb{z}^{(l)}]$ ). We compute the running average of the prediction loss with a window of size  $w$ , expressed by Equation (5), and assume that a boundary is detected when the new prediction loss is higher than the smoothed prediction loss, as expressed by the decision function in Equation (4).
$$
\delta \left(\boldsymbol {x} _ {t} ^ {(l)}; \mu_ {t - 1} ^ {(l)}\right) = \left\{ \begin{array}{l l} 1 & \text {i f} \mathcal {L} _ {\operatorname {p r e d}} \left(\hat {\boldsymbol {x}} _ {t} ^ {(l)}, \boldsymbol {x} _ {t} ^ {(l)}\right) > \mu_ {t - 1} ^ {(l)} \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {4}
$$
where the running average is given by
$$
\mu_ {t} ^ {(l)} = \frac {1}{w} \sum_ {i = t - w} ^ {t} \mathcal {L} _ {\text {p r e d}} \left(\hat {\boldsymbol {x}} _ {i} ^ {(l)}, \boldsymbol {x} _ {i} ^ {(l)}\right) \tag {5}
$$
2.3 Hierarchical gradient normalization
It is necessary to scale the gradients differently from conventional gradient updates because of the hierarchical nature of the model and its learning based on dynamic temporal contexts. There are three variables influencing the amount of accumulation of gradients:
1. The relative timescale between each layer is determined by the number of inputs. For instance, let the event encoder in layer  $(l - 2)$  have seen  $a = |\mathcal{X}^{(l - 2)}|$  inputs, that at layer  $(l - 1)$  have seen  $b = |\mathcal{X}^{(l - 1)}|$  inputs, and that at  $l$  have seen  $c = |\mathcal{X}^{(l)}|$  inputs. Then the input to layer  $l$  is a result of seeing a total of  $(a\cdot b\cdot c)$ . This term can then be used to scale up the learning at any level  $l$ , expressed as  $\prod_{i = 1}^{l}|\mathcal{X}^{(i)}|$ .
2. The reach of influence of each level's representation on a given level's encoder is influenced by its distance from another. For instance, if the input to  $f^{(l)}$  comes from the levels  $\{l + 2, l + 1, l, l - 1, l - 2\}$ , then the weight of learning should be centered at  $l$  and diminish as the distance increases. Such a weight at any level  $l$  is given by  $\alpha^{-|l - r|} \forall r \in \{-2, -1, 0, 1, 2\}$ . To ensure that the learning values sum up to 1 when this scaling is applied, the weights are normalized to add up to 1 as  $\frac{\alpha^{-|l - r|}}{\sum_{i=1}^{L} \alpha^{-|l - i|}}$ .
3. The encoder receives accumulated feedback from predictors of all the layers; therefore the change in prediction loss with respect to encoder parameters in any given layer should be normalized by the total number of layers, given by  $\frac{1}{L}$ .
The temporal encoding model can be learned by scaling its gradients as expressed by the scaled Jacobian  $J_{\mathcal{L}}^{\prime}(\Phi)$  in Equation (6).
$$
\boldsymbol {J} _ {\mathcal {L}} ^ {\prime} (\boldsymbol {\Phi}) = \boldsymbol {C} \circ \boldsymbol {J} _ {\mathcal {L}} (\boldsymbol {\Phi}) = \left[ \begin{array}{c c c} c _ {1, 1} \frac {\partial \mathcal {L} ^ {(1)}}{\partial \boldsymbol {\Phi} ^ {(1)}} & \dots & c _ {1, L} \frac {\partial \mathcal {L} ^ {(1)}}{\partial \boldsymbol {\Phi} ^ {(L)}} \\ \vdots & \ddots & \vdots \\ c _ {L, 1} \frac {\partial \mathcal {L} ^ {(L)}}{\partial \boldsymbol {\Phi} ^ {(1)}} & \dots & c _ {L, L} \frac {\partial \mathcal {L} ^ {(L)}}{\partial \boldsymbol {\Phi} ^ {(L)}} \end{array} \right] \tag {6}
$$
$$
c _ {l, r} = \underbrace {\frac {1}{L}} _ {\text {f e e d b a c k}} \cdot \underbrace {\frac {\alpha^ {- | l - r |}}{\sum_ {i = 1} ^ {L} \alpha^ {- | l - i |}}} _ {\text {r e a c h o f i n f l u e n c e}} \cdot \underbrace {\prod_ {i = 1} ^ {l} \left| \mathcal {X} ^ {(i)} \right|} _ {\text {t i m e s c a l e}} \tag {7}
$$
Similarly, the temporal prediction model's gradients are controlled with scaling factors as expressed in Equation (8).
$$
\boldsymbol {J} _ {\mathcal {L}} ^ {\prime} (\Psi) = \boldsymbol {S} \circ \boldsymbol {J} _ {\mathcal {L}} (\Psi) = \left[ s _ {1} \frac {\partial \mathcal {L} ^ {(1)}}{\partial \Psi^ {(1)}} \quad \dots \quad s _ {L} \frac {\partial \mathcal {L} ^ {(L)}}{\partial \Psi^ {(L)}} \right] \tag {8}
$$
$$
s _ {l} = \underbrace {\frac {1}{\sum_ {i = 1} ^ {L} \alpha^ {- | l - i |}}} _ {\text {r e a c h o f i n f l u e n c e}} \cdot \underbrace {\prod_ {i = 1} ^ {l} \left| \mathcal {X} ^ {(i)} \right|} _ {\text {t i m e s c a l e}} \tag {9}
$$
3.1 Training details