SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chunking Recent work [56] provide a way to chunk data to learn a hierarchical structure by explicitly forming a dictionary of patterns at each level. These methods can be challenging to scale up to real data due to the enormous number of event possibilities to be stored, thus demonstrated only on toy dataset examples of text and vision. Additionally, quantized inputs are required to form these
patterns. In contrast, STREAMAMER does not save patterns but uses learnable networks to encode and predict future inputs. We demonstrate our results on challenging egocentric data.
GLOM and H-JEPA Hinton [26] proposed an idea paper for "GLOM", which attempts to build a part-whole hierarchy by assuming a cortical column for each patch in an image, where each layer in every column receives contributions from nearby columns at higher and lower levels. The hierarchy, or parse tree, is formed by forcing consensus laterally between each level, of each column, using a similarity function. Although we share the same motivation with GLOM, we significantly differ in execution. Instead of using a similarity function to force forming "islands of agreement", STREAMAMER relies on prediction error to discover events and explicitly send their representation to higher level in a dynamically evolving structure (i.e., layer-by-layer). Perhaps we share more algorithmic similarity with LeCun's [37] idea paper, "H-JEPA". The paper proposes to use a hierarchical joint embedding predictive architecture which uses self-supervised predictive learning to predict at different timescales at various levels in the hierarchy. This is also supported by evidence from neuroscience [8] suggesting the importance of long-range forecast representations in improving brain-mapping functionality. H-JEPA suggests that higher levels predict further into the future than lower levels; however, it does not provide a working algorithm for segmentation or cross-layer communication. It is to be noted that both works by Hinton and LeCun only provide ideas and not algorithms/results.
E.3 Predictive modeling
Natural language processing Self-supervised prediction of inputs has shown tremendous success in the field of NLP, specifically masked language models. Recent large language models [13, 5, 52, 54] are based on the core idea of using context to predict missing inputs. The guiding principle of training these models is that successfully predicting missing inputs implies encoding good representation of the context.
Computer vision A similar principle is used to train image representation learning models. Masked Auto Encoder (MAE [25]) is trained to predict missing image patches using a transformer architecture, thus performing better on downstream tasks using the enhanced representations. A family of representation learning methods [21, 6, 9, 7, 43] learn useful representations by predicting augmented views of the input. These augmented views attempt to simulate real-world augmentations (e.g., cropping simulates a moving camera), therefore these methods may also be viewed as predicting future frames in a video sequence. Similar ideas can also be seen applied to video representations learning. PredNet [38] attempts to implement cognitively-inspired predictive coding theories for future frame prediction. Other methods [22, 23] use dense predictive coding to learn video representations for retrieval and fine-tuning on downstream tasks. Unlike STREAMAMER, these methods do not generalize to learning hierarchical representations using predictive learning.
E.4 Relevant datasets
STREAMAMER is a self-supervised architecture that relies on predictive learning for hierarchical segmentation. In our model, higher-order predictive layers receive sparser learning signals than lower-order layers, because the first layer directly predicts frames, whereas higher layers only receive events that cannot be predicted at lower levels. Short videos do not allow for higher order predictions and learning long-term temporal dependencies. Therefore, training higher levels in the hierarchy requires a large dataset (i.e., total number of hours) and longer videos (i.e., average video duration) in order to model high-level events. These requirements constraint the choice of datasets on which we can run and evaluate STREAMAMER.
Based on our review of available datasets for both egocentric and exocentric settings, as shown in the Table 4, many of the available datasets, typically used in event segmentation, do not provide long enough videos. MovieNet and NewsNet are two large datasets with long videos but have not yet been released to the public. In addition, MovieNet does not contain action segments; it contains coarse scene segments. The only available options to train and evaluate STREAMAMER is large-scale egocentric datasets, where the available datasets provide large enough scale (i.e., total number of hours) with a long average duration per video for streaming and high-order temporal prediction.
Table 4: Various egocentric and exocentric datasets with total hours of recording and average duration statistics. * Not released as of this writing.
Dataset	Total Hours	Avg. Duration (min)	Large Datasets	Long Videos
Egocentric	Ego4D	3670	23	✓	✓
EPIC 55	55	10.5	✓	✓
EPIC 100	100	8.5	✓	✓
GTEA	0.58	1.23	✗	✗
Exocentric	Breakfast Action	77	2.3	✓	✗
YouTube Instructional	5	2	✗	✗
Hollywood Extended	3.7	0.23	✗	✗
50 Salads	4.5	4.8	✗	✗
FineGym	708	0.91	✓	✗
ActivityNet Captions	849	2.5	✓	✗
MovieNet*	2174	117	✓	✓
NewsNet*	946	57	✓	✓
F Glossaries
- Contextualized inference In this work, contextualized inference refers to the ability of the model to predict a future event by using contextual representations at various levels of the event hierarchy.
- Contextualized optimization Contextualized optimization refers to the ability of a layer to optimize the representations of other layers through its own prediction loss.
- Event An event is defined as "a segment in time that is perceived by an observer to have a beginning and an end" [59]
- Event model In cognitive psychology literature, an event model is defined as "a representation of what is happening now, which is robust to transient variability in the sensory input" [59]; in this work, we use 'event model' and 'representation' interchangeably.
- Event demarcation, event segmentation Event demarcation is the process of detecting event boundaries by using the prediction loss, whereas event segmentation is the task of segmenting videos (or sensory inputs) into meaningful events. In other words, event segmentation is the goal, and event demarcation is one way to achieve it: event segmentation could be performed in other ways, such as labeling each frame in a supervised framework.
- Predictive learning Predictive learning refers to the brain's ability to generate predictions about future events based on past experiences.
- Segmentation boundary The end of an event and the beginning of the next is a segmentation boundary, marking an event transition.
Table 5: This table shows some examples of the similarity of narrations of 'key events' to a 'query event' as determined by GPT 3.5 (text-davinci-003). A 1 scoring means most similar, and a 0 least.
Query	Keys
open bottle	stop pro-cessor	open freezer	enter kitchen	open door	pick up bowl	open fridge	clean kitchen counter	move cheese	put down sausage	put down pan
0.25	0.8	0.5	0.7	0.4	0.85	0.45	0.3	0.45	0.4
open drawer	pick up stone	check chicken	still cleaning chop- ping board	shake off courgette	rinse pot	pick up	close dish- washer	take out pasta	still scoop the kiwi	pick up something
0.75	0.55	0.65	0.65	0.7	0.8	0.7	0.85	0.65	0.8
wash two leaves	wash knife	put down pepper	rinse pan	put away spoon	wash lid	grab bag	get kettle	pour olive oil in pan	stir cour- gette	pick up forks
0.75	0.45	0.85	0.65	0.95	0.35	0.45	0.75	0.65	0.55
wash pot	wash pan	open jar	open cup- board	pick up mug	pick up aubergine	pour milk into cereal bowl	rinse cloth	lay aubergine	close fridge	open cup- boards
0.85	0.65	0.75	0.8	0.85	0.9	0.85	0.8	0.75	0.75
pick up bowl	turn off tap	put down salt	cut tomato	stir onion	pick up bowl	stir chicken	place pan	grab salt container	open cup- board	pour milk into glass
0.75	0.55	0.45	0.65	1.0	0.7	0.6	0.65	0.5	0.65
get chopping board	move chair	open washing machine door	rinse hands	check oil	pour deter- gent	rinse hands	check tem- perature	put bread onto tray	wash chopping board with sponge	select schedule
0.25	0.5	0.8	0.3	0.5	0.8	0.6	0.7	0.9	0.4
put pot	turn on tap	clean cooker	still rins- ing spatula	put colan- der on pot	put fork in bowl	put lid on pot	put lid on sun-dried tomatoes	rinse hands	wipe down counter	move mouse
0.75	0.45	0.55	0.85	0.65	0.95	0.45	0.55	0.65	0.45
open drawer	take box	pick up jar	close drawer	rinse mug	close hob cover	scrape tomato	lift plate	wash knife	get plastic trash bag	check timer
0.75	0.85	0.95	0.65	0.75	0.65	0.75	0.85	0.65	0.85
close jug	shake cof- fee maker	pick up plate	take nap- kin	get weigh- ing	still taking skin off meat with knife	pick up lid	close tap	get chopping board	put down spoon	put down bowl
0.25	0.5	0.5	0.25	0.05	0.75	0.9	0.5	0.75	0.75
wash bowl	wash cof- fee pot	put down knife	put down detergent	throw away onion skin	turn on tap	rinse chop- ping block	mix nuts with oats	dry hands	place sponge away	open tap
0.95	0.65	0.85	0.45	0.95	0.85	0.65	0.95	0.85	0.95